Systems, methods, and devices for dynamic resource monitoring and allocation in a cluster system

ABSTRACT

In an embodiment, the systems, methods, and devices disclosed herein comprise a computer resource monitoring and allocation system. In an embodiment, the resource monitoring and allocation system can be configured to allocate computer resources that are available on various nodes of a cluster to specific jobs and/or sub-jobs and/or tasks and/or processes.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a continuation application of U.S. patentapplication Ser. No. 14/053,089, filed Oct. 14, 2013 and titled SYSTEMS,METHODS, AND DEVICES FOR DYNAMIC RESOURCE MONITORING AND ALLOCATION IN ACLUSTER SYSTEM, which claims the benefit under 35 U.S.C. 119(c) to U.S.Provisional Application No. 61/841,007, filed Jun. 28, 2013 and titledSYSTEMS, METHODS, AND DEVICES FOR DYNAMIC RESOURCE MONITORING ANDALLOCATION IN A CLUSTER SYSTEM, U.S. Provisional Application No.61/841,074, filed Jun. 28, 2013 and titled SYSTEMS, METHODS, AND DEVICESFOR DYNAMIC RESOURCE MONITORING AND ALLOCATION IN A CLUSTER SYSTEM, U.S.Provisional Application No. 61/841,127, filed Jun. 28, 2013 and titledSYSTEMS, METHODS, AND DEVICES FOR DYNAMIC RESOURCE MONITORING ANDALLOCATION IN A CLUSTER SYSTEM, U.S. Provisional Application No.61/841,025, filed Jun. 28, 2013 and titled SYSTEMS, METHODS, AND DEVICESFOR DYNAMIC RESOURCE MONITORING AND ALLOCATION IN A CLUSTER SYSTEM, U.S.Provisional Application No. 61/841,106, filed Jun. 28, 2013 and titledSYSTEMS, METHODS, AND DEVICES FOR DYNAMIC RESOURCE MONITORING ANDALLOCATION IN A CLUSTER SYSTEM, and U.S. Provisional Application No.61/841,061, filed Jun. 28, 2013 and titled SYSTEMS, METHODS, AND DEVICESFOR DYNAMIC RESOURCE MONITORING AND ALLOCATION IN A CLUSTER SYSTEM. Theforegoing applications are hereby incorporated herein by reference intheir entirety, including specifically but not limited to the systemsand methods relating to dynamic resource monitoring and allocation in acluster computer system.

BACKGROUND

Field

The embodiments of the disclosure generally relate to computer clusters,and more particularly to systems, methods, and devices for the efficientmanagement of resources of computer clusters.

Description of the Related Art

In general, a computer cluster comprises a set of connected computersthat communicate and work together in order to act as a single system. Acomputer cluster can comprise several types of components, including afast local area network, a plurality of computers referred to generallyas nodes, and operating systems running on each node. An advantage ofcomputer clusters is the ability to utilize low cost computer servers inorder to achieve high performance distributed computing that was onlypreviously available through the use of highly expensive main framecomputers. A disadvantage of computer clusters is the increasedoperational challenges that arise when adding more and more nodes to thecomputer cluster. Generally, in order to manage the operationalcomplexities of vast numbers of nodes in a computer cluster, a softwarelayer can be employed to manage the activities of the various computingnodes in order to allow the users to treat the computer cluster as asingle computing unit.

Typically, the software layer for organizing the nodes and orchestratingthe activities on the nodes can be responsible for receiving jobs to beprocessed by the computer cluster. In many instances, the software layerwill divide the job into several tasks or sub-jobs or processes or jobprocesses to be processed by various nodes in the computer cluster.Generally, the software layer is responsible for distributing thesetasks and or sub-jobs or processes or job processes to the availablenodes in the computer cluster. This distribution of tasks or sub-jobs orprocesses or job processes to the various available nodes in a computercluster can lead to performance degradations and/or resourceunderutilization.

SUMMARY

Various embodiments of the present invention relate to the utilizationof computer cluster technology, which generally refers to a plurality ofcomputer servers connected to each other through a fast networkconnection. In an embodiment, the systems, methods, and devicesdisclosed herein comprise a computer resource monitoring and allocationsystem. In an embodiment, the resource monitoring and allocation systemcan be configured to allocate computer resources that are available onvarious nodes of a cluster to specific jobs and/or sub-jobs and/or tasksand/or processes. For example, the system can be configured to controlnetwork utilization across two or more nodes wherein the system canreduce network utilization of a first job that is being performed on afirst node in order to allocate additional network capacity to a secondjob or sub-job that is being performed on a second node. In anotherexample, the system can be configured to reduce the amount of CPU usageon a single particular node that a first job or sub-job is using on thenode in order to allocate additional CPU capacity to a second job orsub-job or process or job process operating on the node.

Generally, the systems and methods herein are configured to processlarge amounts of data received from the various nodes in a cluster inorder to generate, in real time or in substantially real time or on aperiodic basis, instructions for allocating computer resources on thenodes in the cluster. In an embodiment, the system is configured todynamically tune or adjust up or down access to or availability of thecomputer resources provided for on particular nodes in order to ensurethat user-defined goals are satisfied and/or to ensure that the clusteris operating efficiently. In general, the system is configured tocontinuously and/or periodically receive data relating to resourceallocation and/or usage at particular nodes. Additionally, the systemcan be configured to continuously and/or periodically generateinstructions for allocating computer resources at particular nodes forspecific jobs and/or sub-jobs being performed on the nodes of thecluster. The continuous and dynamic changing of resource allocations ona computer cluster in combination with the continuous and/or periodicmonitoring of the resource allocations and/or usage on particular nodesof a cluster results in thousands of transactions over a short period oftime, and makes it impossible for a human being to perform such tasksentirely in a person's mind or by a person using a writing instrumentand paper.

Through the continuous monitoring of the nodes in the cluster andthrough the dynamic allocation of computer resources on particularnodes, the system can be configured to ensure that jobs and/or sub-jobsthat have high prioritization are completed as soon as possible and/orby a user-defined time period. The systems, methods, and devicesdisclosed herein can be utilized in conjunction with specific computercluster types, such as hadoop clusters, or can be configured to operatewith other distributed systems.

In an embodiment, a hadoop computer cluster comprises a master nodecomputing device comprising a management controller and a supervisorcontroller, the management controller configured to coordinate parallelprocessing of data across a plurality of computer system nodes, thesupervisor controller configured to coordinate allocation of systemresources at particular computer system nodes to particular tasks. Theplurality of computer system nodes can be configured to communicate withthe supervisor controller and to perform processing of received tasks.In an embodiment, the computer system nodes each comprise: one or moreprocessors configured to perform computing processes on received tasksand an agent controller. In an embodiment, the agent controller isconfigured to monitor utilization by tasks of system resources of thecomputer system node, the system resources comprising CPU, diskinput/output, network, and memory by the computer system node. In anembodiment, the agent controller is configured to report the monitoredsystem resource utilization to the supervisor in substantiallyreal-time. In an embodiment, the agent controller is configured togenerate instructions for controlling utilization by tasks of systemresources of the computer system node, the instructions based on datareceived from the supervisor controller. The supervisor controller canbe configured to, based on goals specified by an operator of the hadoopcomputer cluster and the substantially real-time reporting from aplurality of agent controllers, determine assignment of tasks torespective computer system nodes and/or resource allocations for eachtask on those nodes such that the operator-specified goals are satisfiedand processing capabilities of the hadoop computer cluster aremaximized. In an embodiment, the management controller comprises a jobtracker. In an embodiment, the management controller comprises a yarnsystem or yarn resource manager.

In an embodiment, a supervisor controller is configured to manage systemresource allocation for a hadoop computer cluster. The supervisorcontroller can comprise a management controller interface configured tocommunicate with a management controller to access data representing anassignment of a plurality of job processes across a plurality ofcomputer system nodes in the hadoop computer cluster, the managementcontroller configured to coordinate parallel processing of data across aplurality of computer system nodes, an agent controller interfaceconfigured to communicate with an agent controller, the agent controllerconfigured to transmit to the agent controller interface datarepresenting utilization of system resources by the plurality of jobprocesses operating on a particular computer system node, a systemresource allocation engine configured to dynamically determine systemresource allocations for particular job processes operating onparticular computer system nodes, the dynamic determination generated bythe system resource engine based on the data representing utilization ofsystem resource by the plurality of job processes operating on theparticular system node; and the agent controller interface configured togenerate data for transmission to the agent controller of a particularcomputer system node based on the dynamic determination generated by thesystem resource engine, the data configured to instruct the agentcontroller to allocate a level of system resources to a particular jobprocess operating on the particular computer system node.

In an embodiment, a computer cluster comprises a management computingdevice comprising a supervisor controller configured to coordinateprocessing of a plurality of sub-jobs for a plurality of overall jobs; aplurality of computer system nodes configured to communicate with thesupervisor controller and to perform processing of received sub-jobs,the computer system nodes each comprising: one or more processorsconfigured to perform computing processes on received sub-jobs; an agentcontroller configured to: monitor utilization of system resources of thecomputer system node, the system resources comprising CPU, electronicstorage input/output, network, and memory by the computer system node;report the monitored system resource utilization to the supervisorcontroller in substantially real-time; wherein the supervisor controlleris configured to, based on goals specified by an operator of thecomputer cluster and the substantially real-time reporting from aplurality of agent controllers, determine assignment of system resourceallocations for each sub-job on those nodes such that theoperator-specified goals are satisfied and processing capabilities ofthe computer cluster are substantially utilized, wherein the managementcomputing device and the plurality of computer system nodes comprise acomputer processor and an electronic storage medium.

In an embodiment, a supervisor controller is configured to manage systemresource allocation for a computer cluster. The supervisor controllercomprises an agent controller interface configured to communicate withan agent controller, the agent controller configured to transmit to theagent controller interface data representing utilization of systemresources by a plurality of sub-jobs operating on a particular computersystem node, a system resource allocation engine configured todynamically determine system resource allocations for particularsub-jobs operating on particular computer system nodes, the dynamicdetermination generated by the system resource engine based on the datarepresenting utilization of system resource by the plurality of sub-jobsoperating on the particular system node; and the agent controllerinterface configured to generate data for transmission to the agentcontroller of a particular computer system node based on the dynamicdetermination generated by the system resource engine, the dataconfigured to instruct the agent controller to allocate a level ofsystem resources to a particular sub-job operating on the particularcomputer system node.

For purposes of this summary, certain aspects, advantages, and novelfeatures of the invention are described herein. It is to be understoodthat not necessarily all such advantages may be achieved in accordancewith any particular embodiment of the invention. Thus, for example,those skilled in the art will recognize that the invention may beembodied or carried out in a manner that achieves one advantage or groupof advantages as taught herein without necessarily achieving otheradvantages as may be taught or suggested herein.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing and other features, aspects and advantages of theembodiments of the invention are described in detail below withreference to the drawings of various embodiments, which are intended toillustrate and not to limit the invention. The drawings comprise thefollowing figures in which:

FIG. 1 is an embodiment of a schematic diagram illustrating a computercluster.

FIG. 2 is an embodiment of a schematic diagram illustrating a computercluster comprising an embodiment of a dynamic monitoring and/or resourceallocation system.

FIG. 2A is an embodiment of a schematic diagram illustrating a computercluster comprising an embodiment of a dynamic monitoring and/or resourceallocation system.

FIG. 2B is an embodiment of a schematic diagram illustrating a computercluster comprising an embodiment of a dynamic monitoring and/or resourceallocation system.

FIG. 3 is a flowchart depicting an embodiment of a process fordynamically monitoring and/or allocating resources across a computercluster.

FIG. 3A is a flowchart depicting an embodiment of a process fordynamically monitoring and/or allocating resources across a computercluster.

FIG. 4 is an embodiment of a schematic diagram illustrating a computercluster comprising an embodiment of a dynamic monitoring and/or resourceallocation system.

FIG. 5 is a flowchart depicting an embodiment of a process formonitoring and/or allocating cluster resources, such as RAM, networkusage, CPU usage, and disk I/O usage.

FIG. 6 is a block diagram depicting a high level overview of anembodiment of a distributor system.

FIG. 7 is a flow chart depicting an embodiment of a process for adistributor as illustrated in FIG. 6.

FIG. 8A is a block diagram depicting a high level overview of anembodiment of virtual clusters.

FIG. 8B is a block diagram depicting a high level overview of anembodiment of virtual clusters.

FIG. 8C is a block diagram depicting a high level overview of anembodiment of virtual clusters.

FIG. 8D is a block diagram depicting a high level overview of anembodiment of virtual clusters.

FIG. 8E is a block diagram depicting a high level overview of anembodiment of virtual clusters.

FIG. 9 is a flowchart depicting an embodiment of a process forprocessing jobs using a virtual cluster.

FIG. 10 is a flowchart depicting an embodiment of a process forprocessing jobs using a virtual cluster.

FIG. 11 is a flowchart depicting an embodiment of a process forprocessing jobs using job groups.

FIG. 12 is a flowchart depicting an embodiment of a process formonetizing and/or budget accounting for resources on a computer cluster.

FIG. 13 is a block diagram depicting a high level overview of anembodiment of a computer cluster comprising heterogeneous nodes.

FIG. 14 is a flowchart depicting an embodiment of a process forprocessing jobs utilizing a heterogeneous computer cluster.

FIG. 15 is a schematic diagram illustrating an embodiment of utilizingjob histories for improving resource allocation of a computer cluster.

FIG. 16 is a flowchart depicting an embodiment of a process forgenerating reports relating to hardware modifications and/or additionsto a computer cluster.

FIG. 17 is a flowchart depicting an embodiment of a process forgenerating reports relating to resource reallocation on a computercluster.

FIG. 17A is a flowchart depicting an embodiment of a process fordetermining resource reallocation levels for application to jobs orsub-jobs.

FIG. 18 is a block diagram depicting a high level overview of anembodiment of a computer cluster comprising a dynamic monitoring and/orresource allocation system.

FIG. 19 is a block diagram depicting an embodiment of a computerhardware system configured to run software for implementing one or moreembodiments of the dynamic monitoring and/or resource allocation systemsdisclosed herein.

DETAILED DESCRIPTION OF THE EMBODIMENTS

Although several embodiments, examples and illustrations are disclosedbelow, it will be understood by those of ordinary skill in the art thatthe inventions described herein extend beyond the specifically disclosedembodiments, examples, and illustrations, and include other uses of theinventions and obvious modifications and equivalents thereof.Embodiments of the inventions are described with reference to theaccompanying figures, wherein like numerals refer to like elementsthroughout. The terminology used in the description presented herein isnot intended to be interpreted in any limiting or restrictive mannersimply because it is being used in conjunction with a detaileddescription of certain specific embodiments of the inventions. Inaddition, embodiments of the inventions can comprise several novelfeatures and no single feature is solely responsible for its desirableattributes or is essential to practicing the inventions hereindescribed.

In general, computer clusters comprise a plurality of computer serversthat are connected to each other through a network connection. In manyinstances, the network connection is a fast network connector such thatall of the computer servers in the cluster can communicate with eachother quickly and efficiently. For example, a computer cluster cancomprise a number of low cost commercially available off-the-shelfcomputers connected through a fast local area network (LAN). In general,a computer cluster can comprise a master node and a plurality of slavenodes. The master node can be configured to coordinate the activities ofthe slave nodes. In an embodiment, the computer hardware for a masternode and for slave nodes are the same or are substantially the same, andare only distinguishable by the assigned roles each computer serverreceives when the cluster has been created. In an embodiment, a clustercan comprise one or more master nodes that coordinate the activities ofvarious slave nodes.

To implement the coordination between the master node(s) and the variousslave nodes, a computer cluster can comprise middleware software thatoperates on each node and that allows communication and coordinationbetween the nodes in order for the computer cluster to act like a singlecohesive computing unit. In general, a master node can be configured todivide jobs and/or processes into smaller jobs and/or processes to beexecuted or processed on one or more slave nodes in order to efficientlyand quickly complete the job. After transmitting a sub-job to a slavenode, a master node generally does not monitor the performance of theprocessing of the sub-job. In some cases, the master node will onlydetermine whether a sub-job has been completed by a designated slavenode.

Accordingly, there are several disadvantages for typical clusterconfigurations. For example, by not verifying or monitoring the statusof a sub-job that is being processed by a slave node, a computer clustersystem may not be able to process a particular job within a time framedesired by the user. Further, by not monitoring and verifying theprogress of a sub-job, the cluster system runs the risk of slowing downhigh priority jobs when the master node adds additional jobs to aparticular slave node. For example, a computer cluster can be configuredto run a job for generating reports on a daily basis. In an embodiment,the computer cluster can be configured to receive additional jobs duringthe period in which the cluster is working on the job for generating theperiodic reports. In such an example, the master node can be configuredto divide the additional job into sub-jobs for further processing byvarious nodes in the cluster. These additional sub-jobs to be processedby the slave nodes can in some instances slow down the completion of thejob for generating the periodic reports.

Without monitoring the progress and/or completion of the job and/or aplurality of jobs for generating the reports, the computer clustercannot determine whether the addition of such ad hoc jobs that are addedto a node are slowing down the time sensitive periodic report generationjob. Accordingly, it can be advantageous for a cluster system to monitorthe completion progress of a particular job and/or a plurality of jobsin order to ensure that such jobs are completed on a timely basispursuant to the specified goals of a user.

Typical computer clusters cannot efficiently handle the addition of adhoc jobs without affecting the performance of jobs that are regularlyscheduled for processing by the cluster. Additionally, typical clusterscannot determine whether a particular node is being overloaded by jobsassigned to the slave node. The overutilization of resources on a slavenode can cause the slave node to experience performance degradations.

For example, if the sub-jobs assigned to a slave node required the useof RAM that exceeds the amount of physical RAM on the node, the slavenode can start to utilize the hard drive to compensation for the lack ofRAM. Writing to a hard drive in order to compensate for the lack of RAMcan cause the slave node to experience significant performance delaysbecause writing to a hard drive is slower than writing to physical RAMor flash memory. The writing to and reading from a hard drive in lieu ofRAM or flash memory can cause severe performance degradations, which cancause “thrashing” of the computer server, requiring the computer serverto be rebooted.

As an example, if multiple sub-jobs assigned to a slave node requestedmore disk I/O accesses per unit time than the node can support, one ormore of the tasks can be slowed down dramatically waiting for disk I/Oaccess. In some cases, the task(s) that may be slowed down could be thehigh-priority regularly scheduled task(s), being slowed down by the adhoc jobs.

Without the active and dynamic monitoring of the resources on a slavenode with respect to the jobs and/or sub-jobs assigned to the slavenode, the computer cluster cannot account for resource overloads on aparticular slave node.

Similarly, without monitoring the resource utilization on the slavenodes within a cluster, the system cannot determine which slave nodesare being underutilized. For example, certain sub-jobs may not requiresignificant amounts of RAM in order to be processed. In certaincircumstances, it can be advantageous for the cluster to assignadditional sub-jobs to the slave node in order to utilize the availableRAM on the slave node. The additional assignment of jobs and/or sub-jobsfor the slave node can ensure that the resources of the slave node arebeing fully utilized.

Typical clusters also do not have the ability to determine which jobs,sub-jobs, processes and/or users are utilizing the cluster to a greaterextent than other jobs and/or users. For example, typical clustersystems cannot determine whether a human resource group is responsiblefor a greater utilization of the cluster relative to a legal departmentof an organization. By not monitoring the resource utilization ofsub-jobs on slave nodes, the cluster system cannot determine how much ofthe resources of the cluster are being utilized by particular jobsand/or users and/or groups of users. It can be advantageous to determinethe percent usage of the cluster by a particular job and/or user and/orgroups of users in order to bill such utilization to a particular joband/or user and/or group of users and/or company department or the like.For example, if the system is configured to determine that a humanresource department utilizes 50% of the resources of the cluster, thesystem can be configured to bill or perform a budgetary accounting thatcauses the human resources department of a company to be responsible for50% of the costs for maintaining the cluster for the company.

Another drawback to typical computer clusters is the system cannotgenerally determine what additional hardware should be added to thecluster in order to efficiently process the jobs and/or sub-jobs beingsent to the cluster for processing. Without monitoring the performanceof jobs and/or sub-jobs being processed by specific slave nodes, thecomputer cluster cannot determine whether bottlenecks exist in thecomputer cluster, wherein the bottlenecks prevent the completion of ajob and/or sub-job in a timely manner. For example, a system that can beconfigured to monitor and determine the resource utilization atparticular slave nodes by particular sub-jobs, can be configured toidentify overutilization of RAM in the cluster. Based on thedetermination that RAM utilization across the cluster is consistentlyabove the available RAM capacity of the cluster, the system can beconfigured to output a message to the user or operator of the cluster toadd additional slave nodes with increased RAM capacity.

In general, typical computer clusters require that the computer serversmaking up the cluster be of the same or similar type of machines.Accordingly, in many instances computer clusters cannot generallycomprise heterogeneous machine types. For example, many computerclusters cannot efficiently operate in an environment where some of thecomputer servers have faster CPU processors than other computers in thecluster. For example, without monitoring the available resources onparticular slave nodes, the cluster system cannot determine that certainslave nodes with faster CPU processors can be configured to take onadditional sub-jobs as compared to other slave nodes in the cluster thathave slower CPU processors that can take on only a limited number ofsub-jobs. Therefore, it can be advantageous for a computer cluster todynamically monitor and allocate resources on a particular slave node inorder to allow a cluster system to fully utilize heterogeneous computerservers in a cluster.

The foregoing shortcomings and disadvantages of typical computerclusters can be addressed by the resource monitoring and allocationsystems disclosed herein. In an embodiment, the system can be configuredto monitor, track, and dynamically control system resources at aper-task/per-process level, an overall per-node level, and an overallper-cluster level in order to maximize the efficiency and/or utilizationof the resources provided for by the nodes in the cluster. The systemresources include but are not limited to CPU usage, RAM usage (bothactual usage and current max limits as set via the virtual machine orkernel), network bandwidth usage, and disk I/O usage (read bandwidth,write bandwidth, and number of disk operations/seeks). In an embodiment,the system can be configured to monitor, track, and dynamically controlat a per-task/per-process level, an overall per-node level, and anoverall per-cluster level several fine-grained resources including butnot limited to:

Disk I/O on a per-device basis; for example, a node with multiplephysical disk drives will generally have read/write bandwidth, seeks,and operations monitored/controlled for each of the physical disk drivesas well as overall.

Network bandwidth broken down by type of access; for example, bandwidthmay be monitored/controlled separately for local rack network access (tothe other nodes sharing the same top-of-rack switch), remote rack access(to other nodes in the same cluster but on a different rack, which canmean using central switch/network bandwidth), and off-cluster access (tonetwork locations outside the cluster, such as an external database orservice).

Distributed filesystem (for example, HDFS) access, which can include acombination of local disk I/O, local rack network, and remote racknetwork. Depending on the kind of access, distributed filesystem usagecan actually take up resources from one or more of the local disk, localrack network, and/or remote rack network. Accordingly, in an embodimentthis distributed filesystem resource needs to be monitored andcontrolled along with direct access to these underlying resources.

Usage of other cluster resources, such as access to the hadoop namenode, and the like.

Usage of off-cluster resources, such as load on an external database,ETL tool, web service, and the like.

In an embodiment, the resource monitoring and allocation systems can beconfigured to work in conjunction with the software middleware of acomputer cluster. For example, the software middleware of the computercluster can be configured to operate normally by receiving jobs from auser, analyzing the received job, dividing the received job intosub-jobs, and distributing the sub-jobs across various slave nodes inthe cluster for processing. The resource monitoring and allocationsystem can complement the activities of the software middleware bymonitoring the jobs and/or sub-jobs being processed on various slavenodes in the cluster.

By monitoring the resource utilization of particular jobs and sub-jobson various slave nodes, the resource monitoring and allocation systemcan be configured to dynamically reallocate resources on particularslave nodes to particular sub-jobs being processed. The reallocation ofresources to particular sub-jobs being processed on particular nodes canallow the computer cluster to operate more efficiently. For example, theresource monitoring and allocation system can be configured toreallocate additional network capacity to high priority sub-jobs inorder for the high priority job to be completed on time. By reallocatingnetwork capacity to high priority sub-jobs, the resource monitoring andallocation system can be configured to slow down the processing ofnon-high priority sub-jobs by reducing the amount of network capacitydedicated to the non-priority sub-jobs.

In an embodiment, the resource monitoring and allocation system cancomprise a supervisor controller system that is configured to monitorthe overall jobs and/or sub-jobs that were initially processed by thesoftware middleware for assignment and processing by the various slavenodes. For example, the supervisor controller can be configured todetermine what resources are being utilized by particular sub-jobsoperating on particular slave nodes. Further, the supervisor controllercan be configured to determine the overall progress in completing anoverall job that has been divided into a plurality of sub-jobs beingprocessed by a plurality of slave nodes. By determining the overallprogress for completing a particular job, the supervisor controller canensure that the overall job is completed to the specifications and/orrequirements set forth by a client and/or user. In order to determinethe particular resource utilization of certain sub-jobs, the resourcemonitoring and allocation system can comprise an agent system.

In an embodiment, the agent system is configured to operate on one ormore of the slave nodes in the computer cluster. In an embodiment, theagent system is configured to operate on each of the slave nodes in acomputer cluster. In an embodiment, the agent system is configured tooperate on a master node. The agent system can be configured todetermine the specific resource utilization at a particular node foreach of the particular sub-jobs. After determining the resourceutilization of a particular sub-job on a particular node, the agentcontroller system can be configured to transmit the resource utilizationdata to a supervisor controller system. In an embodiment, the supervisorcontroller system can be configured to aggregate resource utilizationdata from a plurality of agent controller systems operating on variousnodes in the cluster. The supervisor controller system can be configuredto analyze the resource utilization data to determine the status of thecluster and/or how efficiently the cluster is operating. Further, thesupervisor controller system can be configured to analyze the resourceutilization data to determine whether an overall job is likely to becompleted by the specified time goals set forth by a user of the clusterand/or client.

If the supervisor determines that resources should be reallocated forparticular jobs being processed on particular nodes, the supervisorcontroller system can be configured to generate instructions fortransmission to the agent controller system. The agent controller systemcan be configured to analyze the instructions received from thesupervisor controller system in order to generate specific instructionsfor implementing the resource reallocation on the particular node thatthe agent controller system has control over. Accordingly, the agentcontroller system can serve various roles.

In an embodiment, the agent controller system is configured to monitorresource utilization on a particular node and to determine how eachsub-job being processed on the particular node is utilizing resources ofthe particular node. The agent controller system is also responsible fortransmitting and/or reporting the resource utilization data to thesupervisor controller system. In an embodiment, the agent controllersystem is also responsible for implementing or enforcing the resourcereallocation instructions received from a supervisor controller system.The agent controller system can also be configured to control theallocation of resources to particular jobs and/or sub-jobs that arebeing processed on a particular node. Further, the agent controllersystem can be configured to independently decide whether to reallocateresources of the particular computer node without receiving instructionsfrom the supervisor controller system.

The resources of the node that are being utilized by the system tocomplete the jobs and/or sub-jobs include but are not limited to RAM,CPU capacity, network capacity, and disk I/O capacity. For example, anagent system can be configured to operate on a particular slave nodethat is processing a particular sub-job. The agent system can beconfigured to determine the amount of CPU capacity, RAM capacity,network capacity, and/or disk I/O capacity that is being utilized by theparticular sub-job that is being processed on the particular slave node.

In an embodiment, the system can be configured to obtain the currentresource utilization differently depending on the type of resource. Forexample, the system can be configured to determine CPU capacity bymeasuring actual CPU time used via a call to the kernel and/or readingfiles written by the kernel. In an embodiment, the system can beconfigured to determine RAM capacity by measuring virtual machinestatistics and/or kernel statistics. In an embodiment, the system can beconfigured to determine network capacity by creating a “wrapper” aroundthe code that actually accesses the network, wherein the “wrapper” isconfigured to report statistics of network usage. Alternatively, thesystem can be configured to determine network capacity by using avirtual network interface to intermediate requests to the network,and/or using a “traffic control” command of the kernel or similarkernel-level mechanism to adjust network usage. In an embodiment, thesystem can be configured to determine disk I/O by measuring one or moreof the following: creating a “wrapper” around the code that actuallyaccesses the disk I/O capacity in order to report statistics, and/orusing kernel-level controls to adjust disk I/O usage.

In an embodiment, the agent system can be configured to transmit theresource utilization data for the particular slave node to thesupervisor system. In an embodiment, the supervisor system and/or theagent system can be configured to determine whether a reallocation ofresources should occur at the particular slave node in order to delay oraccelerate the processing of the particular sub-job that is beingprocessed by the particular slave node. For example, the supervisorsystem can be configured to analyze the resource utilization of theparticular sub-job that is being processed by a particular slave nodeand compare the processing performance to other sub-jobs of the sameoverall job being processed by other slave nodes operating within thecomputer cluster.

In an embodiment, the supervisor system can be configured to reallocateadditional CPU capacity to the selected sub-job in order to allow theparticular sub-job to be completed within about the same timeframe asother sub-jobs that are being processed by other slave nodes in thecomputer cluster. By adding the additional CPU capacity to theparticular sub-job, the computer cluster can be configured to preventthe particular sub-job from being a bottleneck in the completion of theoverall job. By removing the bottleneck, the computer cluster system canbe configured to complete the overall job within a user specified timeperiod.

In an embodiment, the agent controller system can be configured todetermine independently from the supervisor controller whether toreallocate resources to a particular sub-job without receiving inputfrom the supervisor controller system. For example, the agent system canbe configured to reallocate additional CPU capacity to a particularsub-job being processed on the particular slave node based ondetermining that the particular sub-job has a higher priority than othersub-jobs being processed by the particular slave node. By addingadditional resource capacity to completing the particular sub-job, theparticular slave node can decrease the amount of processing timenecessary to complete the high priority sub-job. The foregoing examplescan also be applied to other resource types, such as but not limited toRAM capacity, network capacity, disk I/O capacity, and the like.

The supervisor controller system and/or the agent controller system canbe configured to control the allocation of resources on a particularnode through a variety of methods. For example, the agent controllersystem can be configured to control the amount of RAM usage by aparticular sub-job on a particular node by invoking the kill command inan operating system. The kill command is a function that is provided forin a number of commercially available operating systems. The killcommand can be configured to send signals to a running process orprocesses to request the termination of the process. In an embodiment,the agent controller system can be configured to reduce the amount ofRAM utilized by a particular sub-job by sending a kill command to thesub-job thereby eliminating the sub-job's use of any RAM resources inthe node.

Alternatively, the agent controller system can be configured to invokethe JVM (Java Virtual Machine) garbage collection command or othergarbage collection command in order to control the RAM usage for aparticular sub-job. The JVM garbage collection command or other garbagecollection command are generally a form of automatic memory managementthat can be provided for in computer languages, such as Java, C, C++,and the like. In general, garbage collection commands operate by findingdata objects in a program that are no longer in use and by reclaimingthe resources used by the data objects no longer in use, the garbagecollection commands can reduce the amount of RAM usage on a node. In anembodiment, the agent controller system can be configured to control RAMutilization by a particular sub-job by using the garbage collectioncommand to reduce the amount of RAM and/or to recover RAM resources notutilized by the particular sub-job.

In an embodiment, the agent controller system can be configured tocontrol RAM usage for a particular sub-job by adjusting a maximum RAMusage limit function in a virtual machine and/or kernel, and/or byadjusting the number of tasks/processes allowed to run on the nodethrough the virtual machine or kernel. In an embodiment, the agentcontroller system can set the maximum RAM limit for a particular sub-jobbased on the history of similar sub-jobs. For example, if similarsub-jobs have used no more than 500 megabytes of RAM in past runs, themaximum RAM limit for a sub-job can be set to 500 megabytes, instead ofa higher default maximum that is used for sub-jobs in general.

The ability to control RAM is different from the ability to control CPUusage, network usage, and disk I/O usage. For example, an agentcontroller system can be configured to slow down or delay a job and/orprocess in order to reduce or increase the use of network utilization,CPU utilization, and/or disk I/O utilization. However, with respect toRAM, if a program and/or process requires a certain amount of RAM inorder to process a sub-job, the agent controller cannot generallynegotiate with the process in order to reduce the RAM utilizationbecause the required RAM resources are either provided to the sub-job orthe sub-job dies. Accordingly, the agent controller system can beconfigured to either kill a particular sub-job in order to eliminate theRAM utilization by a particular sub-job, or the agent controller systemcan be configured to use the garbage collection functionality in orderto recapture unused RAM by the process or the sub-job.

In an embodiment, the agent controller system can be configured tocontrol the amount of network usage utilized by a particular sub-job ona particular node. The agent controller system can be configured toutilize the sleep command in order to reduce the network utilization bya particular sub-job. The sleep command is provided for in operatingsystems that are commercially available. The sleep command enables aprocess or program to be suspended or delayed for a specific period oftime before the process or program is allowed to execute on the computernode and/or utilize specific resources on the node.

In an embodiment, the agent controller system can be configured tocontrol the network utilization by invoking the sleep command. The sleepcommand will force the sub-job to suspend operations and/or processing,which will in turn suspend and/or delay the network utilization by theparticular sub-job. In an embodiment, the supervisor controller and/orthe agent controller can be configured to generate and/or insert codeinto a sub-job and/or job wherein the code can invoke a sleep call basedon instructions from the supervisor controller and/or the agentcontroller. Alternatively, the agent controller system can be configuredto reduce the network utilization of a sub-job by controlling and/orreducing the bandwidth usage or the amount of bandwidth made availableto a sub-job. In an embodiment, the agent controller system can beconfigured to utilize a traffic shaping utility for controlling thebandwidth that is made available to the particular sub-job. In anembodiment, the system can be configured to control network capacity bycreating a “wrapper” around the code that actually accesses the network,wherein the “wrapper” is configured to control network usage by thecode.

Generally, network utilization is a challenging resource to manage. Forexample, network utilization not only depends upon the amount of networkbeing utilized by a particular sub-job or process, but rather networkutilization also depends upon the amount of network utilization that isbeing used by other sub-jobs and/or processes operating on other partsof the cluster. For example, if a particular first job operating on afirst node is utilizing 60% of the network bandwidth that is availablefor accessing the internet, then a second job being processed by asecond node may only have access to the remaining 40% of the networkbandwidth for connecting to the internet.

The second sub-job operating on the second node can only have access to40% of the network bandwidth notwithstanding the fact that the secondjob can have 100% access to the local area network from the second nodewhere there are no additional jobs that are being processed on thesecond node. Accordingly, in order to monitor and allocate networkresources, the supervisor controller can be configured to receiveresource utilization data from a plurality of agent controller systemsin order to determine an aggregate view of network utilization acrossthe cluster. The global knowledge of network utilization can enable thesupervisor controller to determine which sub-jobs across the clustershould be reduced in order to ensure that a particular sub-job hassufficient network resources available in order to complete the sub-job.

In an embodiment, the agent controller system can be configured tocontrol the amount of CPU usage by a particular sub-job on a particularnode. In an embodiment, the agent controller system can be configured toutilize the nice functionality provided for in an operating system. Thenice functionality is generally provided for in commercially availableoperating systems. The nice command can enable a process and/or sub-jobto have more or less CPU time than other processes or sub-jobs runningon the node. The nice command can allow for assigning differentprocesses and/or sub-jobs with a priority level, and based on thepriority level that has been assigned to the process and/or sub-job, theCPU can provide more or less processing time to the particular processor sub-job. In an embodiment, the agent controller system can beconfigured to reduce the CPU usage of a particular sub-job by assigningthe sub-job a low priority level using the nice command. Alternatively,the agent controller system can be configured to reduce the CPU usage ofa sub-job through the use of cgroups. Generally, cgroups (also known ascontrol groups) provide a mechanism for aggregating and partitioningsets of processes and the future children of the processes into a grouphaving limits on resource utilization. In an embodiment, the agentcontroller system can be configured to utilize cgroups in order to placelimits on the CPU utilization for a particular sub-job that is beingprocessed by a particular node. Alternatively, the agent controllersystem can be configured to reduce the CPU usage of a sub-job throughthe use of posix priorities, a scheduler option built into mostoperating systems, including linux. In an embodiment, the agentcontroller system can be configured to utilize posix priorities in orderto place limits on the CPU utilization for a particular sub-job that isbeing processed by a particular node. In an embodiment, the system canbe configured to control CPU usage by using other kernel mechanisms thatare similar to the nice command, cgroups, and posix priorities describedabove.

The agent controller system can be configured to control the amount ofdisk I/O usage by a particular sub-job that is being processed on aparticular node. In an embodiment, the agent controller system can beconfigured to use at least one of the nice command, cgroups, posixpriorities, or the sleep command in order to reduce the disk I/O usageof a particular sub-job that is being processed by a particular node. Inan embodiment, the system can be configured to control disk I/O bycontrolling one or more of the following: creating a “wrapper” aroundthe code that actually accesses the disk I/O capacity in order tocontrol access to the disk I/O capacity, and/or using kernel-levelcontrols to adjust disk I/O usage.

In an embodiment, the system can be configured to control the usage ofspecific resources, for example, the usage of CPU, RAM, network, anddisk I/O, by controlling the resource through the use of a kernelextension added to the computer operating system, for example a loadablekernel module that is dynamically loaded by the operating system kernel.

In an embodiment, the supervisor controller system can also beconfigured to control the assignment of sub-jobs to particular nodes onthe cluster in order to use resources more efficiently. For example, thesupervisor controller system may determine that a given slave node isrunning primarily sub-jobs that use CPU intensively but do not use RAMor disk I/O intensively, and determine that the given slave node shouldbe assigned additional sub-jobs that require heavy use of RAM or diskI/O but do not require heavy use of CPU.

The various foregoing embodiments of the resource monitoring andallocation system can be implemented and/or utilized in a variety ofcomputer cluster environments. For example, the resource monitoring andallocation system can be implemented in conjunction with a hadoopcluster system. In an embodiment, the resource monitoring and allocationsystem can be implemented in conjunction with non-hadoop clusters, suchas other types of computer clusters configured to operate a variety ofsoftware applications. Software applications include but are not limitedto web servers, databases (for example, MySQL or Impala), virtualmachines, and the like. In an embodiment, the resource monitoring andallocation system can be implemented with other non-hadoop clusters,such as network appliances.

In some versions of the hadoop implementation, the resource monitoringand allocation system can be configured to operate in conjunction withthe job tracker and the task tracker systems. In an embodiment, the jobtracker of the hadoop system divides a new job into a plurality oftasks. The job tracker can be configured to determine the number ofavailable slots or containers in the cluster or in particular nodes toprocess the various generated tasks. The job tracker can be configuredto assign the tasks to various nodes based on the number of slots orcontainers available at a particular node. In an embodiment, the tasktracker of the hadoop system can be configured to transmit to the jobtracker the number of available slots or containers for processingvarious tasks on a particular node. The supervisor controller or theresource monitoring and allocation system can be configured tocommunicate with one or more agent controllers operating on the variousnodes of the cluster. The agent controllers can be configured tocommunicate with the supervisor controller in order to transmit resourceutilization data to the supervisor controller. The resource utilizationdata can include information about how individual tasks are utilizingvarious resources (for example, CPU, RAM, disk I/O, network) of thenode. In an embodiment, the supervisor controller system and/or theagent controller system can be configured to determine whether aparticular task should receive more or less or the same amount of systemresources available at the node that is processing the particular task.

In the context of implementing the resource monitoring and allocationsystem in conjunction with a non-hadoop cluster, the tasks in a hadoopsystem are substituted with software applications and other processes.For example, software applications can include but are not limited toweb servers, databases, virtual machines, and the like. In suchimplementations, the agent controller systems can be configured tooperate on nodes of a cluster and can be configured to monitor theresource utilization of each software application operating on the node.For example, the agent controller system can be configured to determinethe CPU usage, RAM utilization, network usage, and disk I/O usage of aweb server operating on the node.

The agent controller system can be configured to transmit this resourceutilization data to a supervisor controller system. The supervisorcontroller system can be configured to analyze the resource utilizationdata from a plurality of nodes in the cluster to determine whetherresource reallocation is necessary to allow the cluster to operate moreefficiently. The supervisor controller system can be configured totransmit resource reallocation instructions to specific agent controllersystems operating on particular nodes. The instructions can comprisedata necessary for the agent controller system to generate instructionsand/or commands to increase and/or reduce the resource utilization of aparticular software application or other processes that are operating onthe node.

In implementations where the resource monitoring and allocation systemis implemented in a network appliance, such as a network router and/orswitch or the like, an agent controller system can be implemented in thenetwork appliance. In an embodiment, the agent controller system can beconfigured to interrogate the network appliance in order to determinethe resource utilization of particular jobs that are being processed bythe network appliance. For example, an agent controller system operatingon a router and/or switch can be configured to analyze data packets thatare coming into the router and/or switch. In an embodiment, the agentcontroller system can be configured to communicate with a supervisorcontroller system in order to determine which ports of the router and/orswitch through which more data or less data should be processed.

There are many challenges in implementing the resource monitoring andallocation system. Accordingly, one of ordinary skill in the art willappreciate that the systems, methods, and devices disclosed herein forimplementing the resource monitoring and allocation system are novel,unique, and are nonobvious in view of the numerous challenges inimplementing such a system. A challenge in implementing the system isthe automatic tuning of the allocation of resources to various jobs andsub-jobs being processed by plurality of nodes across a cluster. In anembodiment, the automatic tuning of resource allocations in a cluster isbased on desired outcomes inputted into system by the user. For example,a user can define an outcome that is time based. The user can specifythat the project needs to be completed by a certain period of time on aparticular day.

Alternatively, the automatic tuning can be based on a desired resourceallocation as defined by the user. For example, a user of the clustermay define that a particular job must have 75% of the cluster's networkbandwidth capacity as well as 80% of the CPU utilization at a particularnode in the cluster. As another example, a user of the cluster maydefine that a particular job must have access to specifically definedresource minimums, for example at least 100 megabits per second ofnetwork bandwidth, 300 megabytes per second of disk I/O, and 1 billionCPU instructions per second.

The existence of an outcome requirement set by the user can require theresource monitoring and allocation system to have access to globalknowledge of the cluster in order to properly monitor and control thevarious nodes such that the user defined outcomes can be achieved. Forexample, the resource monitoring and allocation system must globallydetermine and globally control the network usage of each node in thecluster in order to ensure that 75% of the network bandwidth capacity isdedicated to the particular job or sub-job designated by the user. Thiscan require that the resource monitoring and allocation system reducethe network utilization of certain jobs or sub-jobs operating on othernodes of the cluster in order to provide excess network bandwidth to theparticular job or sub-jobs that the user required to have 75% of thenetwork bandwidth of the cluster.

In an embodiment, the resource monitoring and allocation system can beconfigured to identify jobs or sub-jobs that have been allocated acertain amount of computer resources but is only utilizing small portionof the resource allocation. By identifying such jobs or sub-jobs, thesystem can be configured to re-allocate a portion of the resourceallocation to another job or sub-job. For example, the system can beconfigured to identify a first sub-job that is being processed by afirst node, wherein the first sub-job has been allocated 75% of thenetwork resource capacity but is only utilizing 25% of the networkresource capacity. The system can be configured to reallocate a portionof the network resource capacity from the first sub-job to a secondsub-job that is being processed on the first node or another node.Further, the system can be configured to reallocate the portion of thenetwork resource capacity from the second sub-job back to the firstsub-job if the system identifies that the performance of the firstsub-job declines due to a lack of network resource capacity.

Determining the available resources across a computer cluster can bechallenging because the status of the cluster is continuously changing.Therefore, the resource monitoring and allocation system requirescontinuous updated information regarding the resource utilization ateach node in the cluster. As the information about the status of thevarious nodes in the cluster changes the resource monitoring andallocation system can be configured to adapt accordingly. Anotherchallenge of the resource monitoring and allocation system is themanaging, processing, analyzing, and logging of the large amount of datatransmitted to the supervisor controller from the plurality of agentcontrollers operating in the various nodes of the cluster. In anembodiment, the resource monitoring and allocation system can beconfigured to receive resource allocation data from each node in thecluster once every 1 second to 5 seconds. The sheer volume of datacoming into the monitoring and allocation system makes it impossible fora human being, whether entirely in the person's mind or whether theperson is using a pen and paper, to track and/or perform, in real-timeor substantially real-time, the activities of the embodiments of theresource management and allocation systems that are disclosed herein.

FIG. 1 is an embodiment of a schematic diagram illustrating a computercluster. In an embodiment, the computer cluster 101 can comprise amaster node 104 connected to a network 108. The computer cluster 101 canalso comprise a plurality of nodes 110, 120, 130 that are connected toeach other and to the master node 104 through network 108. In anembodiment, the cluster 101 can be configured to communicate with client102. The master node 104 can be configured to receive from the client102 jobs for processing on the cluster 101. In an embodiment, the masternode 104 can be configured to return to the client 102 completed jobsthat have been processed by the cluster 101.

The master node 104 can be configured to analyze jobs received from theclient 102. The master node 104 can be configured to divide the jobreceived from the client 102 into a plurality of smaller jobs orsub-jobs. The master node 104 can be configured to distribute and/orassign the smaller jobs or sub-jobs to various nodes 110, 120, 130 inthe cluster 101. In assigning the smaller sub-jobs to the various nodes110, 120, 130, the master node 104 may be configured to utilizemanagement software 106 for managing and/or tracking the smaller jobsthat have been distributed across the cluster 101.

In an embodiment, the management software 106 is implemented using ahadoop system. In a hadoop system, the management software 106 cancomprise software known as job tracker. Alternatively, the managementsoftware 106 can be implemented using the Yarn software or Yarn resourcemanager and/or Yarn node manage in a hadoop system. In non-hadoopsystems, the management software 106 can comprise other softwareapplications that are configured to analyze jobs, divide jobs intosmaller sub-jobs, and/or distribute the sub-jobs to various nodes in thecluster 101 for processing.

In an embodiment, the slave nodes 110, 120, 130 can comprise software112, 122, 132 for tracking the sub-jobs that are being processed on thenode. In an embodiment, the nodes 110, 120, 130 can comprise a storagedevice 118, 128, 138 configured to store data and/or software forprocessing the sub-jobs received from the master node 104. In anembodiment, the software 112, 122, 132 is configured to track sub-jobs114, 116, 124, 126, 134, 136 that have been received from the masternode 104 for further processing on the node. In an embodiment, thesoftware 112, 122, 132 can be configured to communicate with the storagedevices 118, 128, 138 in order to process the sub-jobs.

FIG. 2 is an embodiment of a schematic diagram illustrating a computercluster comprising an embodiment of a dynamic monitoring and/or resourceallocation system. In an embodiment, a cluster 201 can be configured tocommunicate with a client 202. The client can be configured to send ajob for processing on the cluster 201. The cluster 201 can be configuredto return a completed job to the client 202. In an embodiment, thecluster 201 can comprise a master node 204 as well as a plurality ofslave nodes 210, 232. The master node 204 can be configured to analyzethe job received from client 202. The master node 204 can comprisesoftware 206 for analyzing the job, dividing the job into sub-jobs,and/or distributing the sub-jobs to the various slave nodes in thecluster 201. In a hadoop system, the software 206 can comprise the jobtracker software or the Yarn software. In non-hadoop systems, thesoftware 206 can comprise other management software for analyzing jobs,dividing jobs into sub-jobs, and/or distributing sub-jobs across thecluster to various nodes.

In an embodiment, the software 206 can be configured to divide the jobinto four sub-jobs 212, 214, 228, 230. In a hadoop system, the sub-jobsare known as tasks. In non-hadoop systems, the smaller jobs that aregenerated by the master node 204 are generically known as sub-jobs. Asillustrated in FIG. 2, the management software 206 can be configured todistribute sub-jobs 214 to a first node 210 and can be configured todistribute sub-jobs 228, 230 to a second node 232.

In an embodiment, the slave nodes 210, 232 can comprise software 216,234 for tracking sub-jobs that have been assigned to a particular node.In a hadoop system, the software 216, 234 can comprise the task trackersoftware. In non-hadoop systems, the software 216, 234 can compriseother node manager software for tracking the sub-jobs that have beenassigned to a particular node from a master node 204.

In an embodiment, the master node can comprise a supervisor controller208. The supervisor controller 208 can be configured to monitor, track,log, and/or control the allocation of computer resources at particularnodes 210, 232. In an embodiment, the nodes 210, 232 can comprise anagent controller 218, 236. The agent controller 218, 236 can beconfigured to monitor, track, log and/or control the allocation ofcomputer resources on a particular node. For example, the agentcontroller 218 can be configured to communicate with the kernel of thenode or other systems on the node 220 to determine the computerresources being utilized by the sub-jobs 222, 224 that are operating onnode 210.

In determining the resource utilization of particular sub-jobs operatingon a node, the agent controller 218, 236 can be configured to transmitthe resource utilization data to the supervisor controller 208. In anembodiment, the supervisor controller 208 can be configured to analyzethe resource utilization data received from the agent controller 218,232 in order to determine whether computer resources that are currentlybeing utilized by certain sub-jobs should be reallocated to othersub-jobs. Based on the foregoing determination, the supervisorcontroller 208 can be configured to generate instructions fortransmission to the agent controller 218, 236. The instructions can beconfigured to cause the agent controller 218, 236 to generate furthercommands to control the allocation of resources on a particular node210, 232 for use by various sub-jobs 222, 224, 240, 242.

In an embodiment, the agent controller 218, 236 can be configured togenerate commands for controlling the allocation of resources on aparticular node without receiving instructions from a supervisorcontroller 208. For example, an agent controller 218, 236 can beconfigured to increase and/or decrease CPU capacity directed to aparticular sub-job 222, 224, 240, 242 based on the prioritization of thesub-task. In an embodiment, the agent controller 218 can be configuredto determine that the sub-job 222 has a higher priority than that ofsub-job 224. Based on the foregoing determination, the agent controller218 can be configured to increase the CPU capacity directed to sub-job222 while decreasing the CPU capacity for sub-job 224. In an embodiment,the foregoing reallocation of computer resources can be performed by theagent controller 218 without instructions from the supervisor controller208.

FIG. 2A is an embodiment of a schematic diagram illustrating a computercluster comprising an embodiment of a dynamic monitoring and/or resourceallocation system. Similar to FIG. 2, a client 202 can submit jobs forprocessing on cluster 201 a. In an embodiment, cluster 201 a cancomprise a master node 204 and a slave node 246. In contrast to FIG. 2,the cluster 201 a as illustrated in FIG. 2A can comprise a supervisorcontroller 208 that operates on node 246 while a job tracker or othermanagement software 206 operates on master node 204.

The advantage of separating the job tracker or other management software206 from the supervisor controller 208 is to ensure that the job trackeror other management software 206 has sufficient computer resources onthe master node for processing the job submissions received from client202. Similarly, by positioning the supervisor controller 208 on aseparate node 246, the operator of the cluster 201 a can ensure that thesupervisor controller has sufficient computer resources dedicated to thesupervisor controller 208 such that the supervisor controller 208 cancontinuously monitor, process, and/or analyze all of the resource datathat is being received form the plurality of agent controllers 218, 236.

Additionally, by positioning the supervisor controller 208 on a separatenode 246, the operator of the cluster 201 a can ensure that thesupervisor controller 208 has sufficient computer resources fordynamically and automatically generating instructions for controlling inreal time or substantially real time the allocation of resources on aparticular node for a particular task operating on the node.

FIG. 2B is an embodiment of a schematic diagram illustrating a computercluster comprising an embodiment of a dynamic monitoring and/or resourceallocation system. Similar to FIGS. 2 and 2A, a client 202 can submitjobs for processing on cluster 201 b. In contrast to FIGS. 2 and 2A, thecluster 201 b as illustrated in FIG. 2B can comprise a first supervisorcontroller 208 that operates on node 246 and a second supervisorcontroller 209 that operates on node 254. As illustrated in FIG. 2B, thejob tracker or other management software 206 is positioned on masternode 204.

The advantage of this configuration is the ability to ensure that thenecessary computer resources are being allocated to the supervisorcontroller systems 208, 209 and the job tracker or other managementsoftware 206. In an embodiment, the first supervisor controller 208 andthe second supervisor controller 209 can be configured to communicatewith different agent controllers 218 and 236. For example, the firstsupervisor controller 208 can be configured to communicate with agentcontroller 218 while the second supervisor controller 209 can beconfigured to communicate with agent controller 236. In an embodiment,the agent controllers 218 and 236 communicate only with predesignatedsupervisor controllers 208, 209. For example, the agent controller 218can be configured to only communicate with supervisor controller 208while the agent controller 236 can be configured to only communicatewith supervisor controller 209.

In an embodiment, the agent controllers 218, 236 can be configured tocommunicate with the supervisor controllers 208, 209 on a first come,first served basis. For example, the agent controller 218 can beconfigured to communicate with either the first supervisor controller208 or the second supervisor controller 209 depending upon whichsupervisor controller is available at any particular time. Similarly,the agent controller 236 can be configured to communicate with eitherthe first supervisor controller 208 or the second supervisor controller209 depending upon which supervisor controller is available at any oneparticular time.

The advantage of comprising two or more supervisor controllers in acluster system is to ensure that the supervisor controllers havesufficient computer resources to continuously monitor, track, analyze,log, and/or control the allocation of computer resources on a particularnode for any particular sub-job operating on a node. In an embodiment,the two or more supervisor controllers 208, 209 can be configured tocommunicate with each other in order to share tracking informationrelated to the allocation of computer resources across various nodes inthe cluster. The two or more supervisor controllers 208, 209 can beconfigured to communicate with each other in order to coordinate thecontrol of the allocation of computer resources at particular nodes inthe cluster.

FIG. 3 is a flow chart depicting an embodiment of a process fordynamically monitoring and/or allocating resources across a computercluster. In an embodiment, the process can start at block 302 with aclient submitting a job or other submission to the hadoop system. Atblock 304, the job tracker of the hadoop system can be configured toreceive the submission from the client. At block 306, the job trackercan be configured to invoke the map reduce function in the hadoop systemto use map process in order to divide the submission into various tasks.At block 306, the job tracker can be configured to invoke the map reducefunction of the hadoop system in order to assign the task to variousslave nodes in the cluster.

At block 308, the slave nodes are configured to receive the assignedtask from the job tracker. In an embodiment, the slave node comprises atask tracker that is configured to receive the task from the jobtracker. At block 310, the slave node can be configured to process thetask received from the job tracker. At block 312, the task tracker canbe configured to determine if the task has terminated or failed duringthe processing by the node. If the task has terminated or failed, atblock 314, the slave node informs the job tracker. At block 314, the jobtracker reassigns the terminated or failed task to another slave nodeand returns to block 308. If at decision block 312, the task has notterminated, the system moves to block 316.

At block 316, the agent controller that is operating on the nodeperiodically or continuously accesses or interrogates the slave node toobtain computer resource data from the kernel or other modules. In anembodiment, the agent controller at block 316 can be configured to trackthe task in the slave node. At block 320, the agent controller can beconfigured to transmit the computer resource status data to thesupervisor controller. While the node is processing the task that hasbeen assigned to the node at block 310, the agent and supervisorcontrollers can be configured to track the assigned task at block 318.

At block 322, the agent and/or supervisor controllers periodically or inreal time determine whether the computer resources that are beingallocated to each task at each particular node should be changed. In anembodiment, the system can be configured at block 324 to generateinstructions for the slave node to dynamically change the allocation ofresources being utilized by particular jobs on a particular node if theagent and/or supervisor controllers determine that the computer resourceallocation is above a threshold level for a particular task operating ona particular node. For example, the agent and/or supervisor controllerscan be configured to determine that a particular job is utilizing RAMthat exceeds a threshold limit or level for a particular node. Inresponse, the agent and/or the supervisor controllers can be configuredto instruct the node to terminate the job if the job is utilizing RAMthat exceeds a threshold limit or level for a particular node.

At block 312, the task tracker can be configured to determine that thetask has been terminated and inform the job tracker at block 314. Atblock 314, the job tracker can be configured to reassign the terminatedtask to another slave node. In an embodiment, the supervisor can beconfigured to use the historical data relating to the previoustermination of the job in order to instruct the job tracker to assignthe previously terminated task to a node having enough RAM capacity toallocate to the job, thereby preventing the job from being terminatedagain. Alternatively, the supervisor controller can be configured todirectly assign the previously terminated task to a node having enoughRAM capacity to allocate to the job, thereby avoiding the need for thejob tracker to assign the task to a new node.

If the agent and/or supervisor controllers determine that the job isoperating within an acceptable range or is below a particular thresholdlevel, then the system can be configured to return the block 312 todetermine if the task has died or terminated. If the process has notbeen terminated the system continues to block 316 to periodically orcontinuously access the computer resource status data on a particularnode.

FIG. 3A is a flow chart depicting an embodiment of a process fordynamically monitoring and/or allocating resources across a computercluster. Similar to FIG. 3, the agent controller can be configured toperiodically or continuously access the computer resource status data ona particular slave node. At block 320, the agent controller can beconfigured to transmit the computer resource status data to thesupervisor controller. While the slave node processes the task at block310, the agent and/or supervisor controller at block 326 can beconfigured to track the task on a particular node and determine thepriority of the task based on client input when the job was submitted tothe job tracker.

At block 328, the agent and/or supervisor controllers periodically or inreal time determine the resources to be allocated to each task on aslave node based optionally on the prioritization of the task asdetermined by the client or based optionally on whether the jobperformance is below a minimum performance guarantee specified by theclient. At block 324, the system can be configured to determine if aresource allocation is above a threshold level for a particular taskand/or node or if a job is operating below a designated priority levelor if the job performance is below a minimum performance guarantee, thenthe system can be configured to generate instructions for the slave nodeto dynamically change the allocation of computer resources to bededicated to the job in order to bring down the resource allocationbelow a threshold level, or to ensure that the job is operating at aspecific priority level or to ensure that the job performance is above aminimum performance guarantee.

FIG. 4 is an embodiment of a schematic diagram illustrating a computercluster comprising an embodiment of a dynamic monitoring and/or resourceallocation system. Similar to FIGS. 2, 2A, 2B, a client 402 cancommunicate with one or more master nodes or other nodes 404 in order tosubmit a job for processing on a computer cluster 401. In an embodiment,the master node 404 can comprise a management software 406 and asupervisor controller 408. In an embodiment, the supervisor controller408 and the management software 406 operate on a single master node 404.In an embodiment, the supervisor controller 408 and the managementsoftware 406 operate on separate master nodes 404. In an embodiment, thejob that is submitted by client 402 is received by the managementsoftware 406 that is responsible for analyzing the job and dividing thejob into smaller sub-jobs. As illustrated in FIG. 4, the system can beimplemented in conjunction with a hadoop system; however, one ofordinary skill in the art will appreciate that the systems and methodsdisclosed herein can be used in conjunction with other cluster systemsand not just with hadoop systems.

The divided sub-jobs 414, 410, 438, 440 can be assigned by themanagement software 406 to various nodes 416, 442 in the cluster. In anembodiment, a node manager (or a task tracker in a hadoop system) 418,444 can be configured to receive the sub-jobs that have been assigned toa particular node by the management software 406. The supervisorcontroller 408 can be configured to communicate with the agentcontrollers 420, 446 that operate on the nodes 416, 442 of the cluster.While the nodes 416, 442 are processing the sub-jobs, the node manager418, 444 can be configured to track the sub-jobs being operated on byparticular nodes.

Additionally, the agent controllers 420, 446 can be configured to alsotrack the sub-jobs being operated by the nodes 416, 442 in addition todetermining the allocation of computer resources to each of the sub-jobson a particular node. For example, agent controller 420 can beconfigured to communicate with the kernel or other module 422 of thenode 416 in order to determine the amount of network capacity 424, RAMusage 426, disk I/O usage 428, and CPU capacitor 430 as being utilizedby the sub-jobs 432, 434 that are being operated on by the node 416. Theagent controller 420 can be configured to transmit the computer resourceallocation data to the supervisor controller 408. The agent controller420 and/or the supervisor controller 408 either alone or in conjunctionwith each other, can be configured to determine which sub-jobs 432, 434are utilizing acceptable allocations of computer resources of the node416. For example, the agent controller 420 can be configured todetermine that a first sub-job 432 is utilizing excess disk I/O capacity428.

In an embodiment, the foregoing determination can be based on theprioritization assigned to the first sub-job 432. If the sub-job 432 hasa low prioritization but is utilizing substantially all of the disk I/Ocapacity 428, the agent controller 420 can be configured toindependently reduce the amount of disk I/O capacity 428 that isallocated to the sub-job 432 in order to provide the second sub-job 434greater access to the disk I/O capacity 428.

In another example, the supervisor controller 408 and the agentcontrollers 420, 446 can be configured to coordinate with each other inorder to collectively determine and/or control the resource allocationsthat are provided to various sub-jobs operating on the nodes 416, 442.In an embodiment, the supervisor controller 408 can be configured todetermine that the third sub-job 458 is utilizing 100% of the networkcapacity 450 by analyzing the resource data transmitted to thesupervisor controller 408 from the agent controller 446.

In an embodiment, the 100% utilization of the network capacity 450 canresult in the 100% network capacity utilization for the entire cluster401. Accordingly, the first sub-job 432 operating on node 416 comprises0% of the network capacity 424 for node 416 to process the sub-job 432.In an embodiment, the first sub-job 432 comprises a high priority ratingwhereas the third sub-job 458 comprises a low priority rating. Thesupervisor controller 408 can be configured to generate instructions forinstructing the agent controller 446 to reduce the network capacity 450that is allocated to the third sub-job 458. The supervisor controller408 can also be configured to instruct the agent controller 420 toprovide additional network capacity 424 to the first sub-job 432.

FIG. 5 is a flow chart depicting an embodiment of a process formonitoring and/or allocating cluster resources, such as RAM, networkusage, CPU usage, and disk I/O usage. The process can start at block 502with the agent and/or supervisor controllers accessing the statusupdates from the slave nodes. At block 504, the agent and/or supervisorcontrollers can be configured to determine if the RAM usage is above athreshold level at a particular node for a particular job. If thedetermination is yes, at block 506, the agent and/or supervisorcontrollers can be configured to determine a mechanism to reduce the RAMusage for a particular task on a particular node. For example, the agentand/or supervisor controller can be configured to optionally kill a taskin order to reduce the RAM usage for a particular task.

In an embodiment, the agent and/or supervisor controllers can beconfigured to optionally kill low priority sub-jobs in order to free RAMusage for other high priority jobs operating on the same node. The usageof RAM, unlike other computer resources, is difficult to reduce or limitfor a particular task. Generally, a job will require a certain amount ofRAM to operate and if the job does not receive the required RAM usage,then the job cannot be performed. Accordingly, there is less discretionin controlling RAM usage as compared to controlling network usage, CPUusage, and disk I/O usage. Alternatively, the agent and/or supervisorcontrollers can be configured to optionally invoke the garbagecollection command of an operating system. For example, the agent and/orsupervisor controller can be configured to invoke the JAVA virtualmachine garbage collection command for a particular task in order toreduce the RAM usage by that task on a particular node.

If at block 504, the agent and/or supervisor controllers determine thatthe actual RAM usage is below a threshold level at a particular node,the agent and/or supervisor controllers at block 508 can be configuredto determine whether additional tasks should be assigned to the node. Ifthe determination is yes, then at block 512, the supervisor controllercan be configured to instruct the management software 206 (for examplethe job tracker in a hadoop system) to assign new tasks to the slavenode. Alternatively, at block 512, the supervisor controller can beconfigured to assign a new task to the slave node without instructingthe management software 206. If at block 508 the determination is no,the system at block 516 has determined that historically such tasks ofthis type use excess RAM.

At block 518, the agent and/or supervisor controllers can be configuredto determine if the network usage is above a threshold level at aparticular slave node for a particular job. If the determination is yes,at block 520, the agent and/or supervisor controllers can be configuredto determine a mechanism for reducing the network usage. For example,the agent and/or supervisor controllers can be configured to optionallysleep a task at block 524. Alternatively, the agent and/or supervisorcontrollers can be configured to optionally reduce bandwidth usage atblock 526.

If the determination at block 518 is no, the agent and/or supervisorcontroller can be configured to determine if additional tasks should beassigned to the node. If the determination is yes, the agent and/orsupervisor controllers can be configured to assign at block 528additional tasks to the node and/or allow a current task more networkaccess. If the determination is no at block 522, the agent and/orsupervisor controllers have made a determination that historically suchtasks of this type use excess network capacity and therefore noadditional tasks should be assigned to this node.

At block 532, the agent and/or supervisor controllers can be determinedif CPU usage is above a threshold level at a particular node for aparticular task. If the determination is yes, the agent and/orsupervisor controllers can be configured to determine a mechanism toreduce the CPU usage for a particular task. For example, the agentand/or supervisor controllers can be configured to optionally “nice” atask. Alternatively, the agent and/or supervisor controllers can beconfigured to optionally invoke a Cgroup command for a task in order toreduce the CPU usage for a particular task.

If the determination is no at block 532, then the agent and/orsupervisor controllers can be configured to determine if additionaltasks should be assigned to the node. If the determination is yes, atblock 540 the supervisor controller can be configured to instruct themanagement software 206 to assign a new sub-job to the slave node.Alternatively, the supervisor controller can be configured to directlyassign a new sub-job to the node. If the determination is no at block536, then at block 544 the agent and/or supervisor controllers have madea determination that historically the job of this type uses excess CPUand therefore no additional sub-jobs should be assigned to this node.

At block 546, the agent and/or supervisor determines if disk I/O usageis above a threshold level at a particular slave node. If thedetermination is yes, then at block 548 the agent and/or supervisorcontrollers determine a mechanism to reduce the disk I/O usage for aparticular task. For example, the agent and/or supervisor controllerscan be configured to optionally nice, Cgroup, or sleep a particularsub-job at block 552. If the determination is no at block 546, the agentand/or supervisor controllers can be configured to determine ifadditional sub-jobs should be assigned to the node. If the determinationis yes, then at block 554 the supervisor controller and/or themanagement software 206 can be configured to assign a new task to theslave node. If the determination is no at block 550, then at block 556,the agent and/or supervisor controllers have made a determination thathistorically such sub-jobs of this type use excess disk I/O andtherefore no additional sub-jobs should be assigned to this node.

FIG. 6 is a block diagram depicting a high-level overview of anembodiment of a distributor system. In an embodiment, a supervisorcontroller, an agent controller, a disk, a network appliance, or otherdevice 602 that is in a cluster or connected to a cluster can comprise adistributor 604. In an embodiment, a distributor 604 can be configuredto receive a variety of inputs in order to determine the resourceallocations for a particular task operating on a particular node. In anembodiment, the distributor 604 can be configured to receive data 606regarding the state of a node and/or the computer resource usages at aparticular node.

The distributor 604 can be configured to analyze the data 606 in orderto generate limits and/or allocations of various computer resources fora particular task on a particular node. The limits and/or allocations ofvarious computer resources can be generated as outputs 612 by thedistributor 604 wherein the output 612 can be utilized by the supervisorcontroller, agent controller, disk, network appliance, or other device602 in order to generate instructions for adding or reducing theallocation of computer resources to a particular job or sub-job.

In an embodiment, the distributor 604 can be configured to receive as aninput 608 operator specified goals and/or properties for a particularjob and/or sub-job. For example, an operator, or client, or other userof a cluster system can specify that a job be completed in a less than aspecified period of time or that a job must be provided a minimum levelof network access in order to complete the job. In an embodiment, thedistributor 604 can be configured to analyze the operator specifiedinputs in order to generate an output 612 for limiting and/or allocatingvarious computer resources for a particular task operating on aparticular node.

In an embodiment, the distributor 604 can be configured to receivehistorical data inputs. In an embodiment, historical data inputs caninclude data relating to how similar jobs of this type require specificCPU usages, RAM usages, network usages, and/or disk I/O usages. In anembodiment, the distributor 604 can be configured to analyze thehistorical data inputs 610 in order to generate outputs 612 relating tolimitations and/or allocations of various computer resources forparticular jobs or sub-jobs on particular nodes.

FIG. 7 is a flow chart depicting an embodiment of a process for adistributor as illustrated in FIG. 6. In an embodiment, the process canbegin at block 702 with the system accessing data at block 704. The datacan be related to the state of a cluster(s) and/or parts of a clusterand/or external resources. For example, the system can be configured toaccess computer resource usage data relating to a particular joboperating in a particular node. At block 706, the system can beconfigured to access data relating to operator(s) specified goal(s)and/or performance properties for a particular job.

At block 708, the system can be configured to access data relating tohistorical research requirements for similar jobs and/or tasks. In anembodiment, the system can be configured to optionally access historicaldata relating to historical resource requirements for similar jobsand/or tasks that are performed on particular or similar nodes. At block710, the system can be configured to optionally access priority datarelating to job submissions in process or in queue to determine globalpriority of job submissions relative to each other. At block 712, thesystem can be configured to analyze the data inputs to determine limitsand/or resource allocations for particular jobs and/or tasks operatingon particular nodes.

At block 714, the system can be configured to generate instructions forlimiting and/or allocating resources for particular jobs and/or tasksthat are operating on particular nodes. At block 716, the system can beconfigured to transmit the instructions to cluster(s) and/or node(s) orsub-jobs or external resources. At block 716, the process can beconfigured to end or it can be configured to optionally return to block704 to repeat the process.

FIG. 8A is a block diagram depicting a high-level overview of anembodiment of virtual clusters. In an embodiment, the client 802 can beconfigured to submit jobs to a virtual cluster. As illustrated, client802 can be configured to submit a job to a master node 804. The masternode can comprise a job tracker or other management software 806 and asupervisor controller 808. In an embodiment, the job tracker or othermanagement software 806 can be configured to analyze the job receivedfrom the client 802 in order to divide the job in to a plurality ofsub-jobs for distribution and processing by various nodes in thecluster.

In an embodiment, the supervisor controller 808 can be configured todetermine whether the job received from the client 802 should beprocessed on a first virtual cluster 805 or whether the job should beprocessed on a second virtual cluster 807. As illustrated in FIG. 8A,there is only one physical cluster for processing the job that isreceived from client 802. However, the supervisor controller 808 can beconfigured to dynamically create one or more virtual clusters from onephysical cluster. For example, the supervisor controller 808 can beconfigured to allocate nodes 1, 2, and 3 to form a first virtual cluster805 dedicated to processing certain jobs of the client 802 and thesupervisor controller 808 can be configured to designate nodes 4, 5, and6 as a second virtual cluster 807 that is dedicated to processinganother type of job received from client 802.

The advantage of creating virtual clusters is an operator need notcreate separate physical clusters in order to have dedicated clustersfor processing certain client jobs. Rather, the operator needs only onecluster that can be divided into one or more virtual clusters that arededicated to certain client jobs. The advantage of virtual clusters overmultiple physical clusters is operational simplicity. The operator needonly maintain one physical cluster as opposed to multiple physicalclusters. In an embodiment, the supervisor controller 808 can beconfigured to analyze the sub-jobs and/or the job submitted by theclient 802 in order to determine which virtual cluster should processthe job and/or sub-jobs.

In an embodiment, the supervisor controller 808 can be configured todetermine that the job submitted by the client 802 is a high priorityjob. For high priority jobs, the supervisor controller 808 can beconfigured to submit the related sub-jobs to the second virtual cluster807, which can process the sub-jobs faster because the nodes in thesecond virtual cluster 807 have been allocated with 75% CPU capacity. Incontrast, the supervisor controller 808 can be configured to determinethat a client job is a low priority job and therefore should be assignedto the first virtual cluster 805, which will process the sub-job slowerthan the second virtual cluster 807. The reason why the first virtualcluster will process the sub-job slower is because the nodes in thefirst virtual cluster 805 have only been allocated 50% of the CPUcapacity of each node.

FIG. 8B is a block diagram depicting a high-level overview of anembodiment of virtual clusters. Similar to FIG. 8A, the client 802 canbe configured to submit jobs to the master node 804. In contrast to FIG.8A, the supervisor controller 808 can be configured to dynamicallygenerate virtual clusters. As illustrated in FIG. 8B, the supervisorcontroller 808 initially created a first virtual cluster 824 comprisingnode 1, 810, node 2, 812, and node 3, 814. The supervisor controller 808can be configured to dynamically generate a new first virtual cluster826. The dynamic generation of virtual clusters can be advantageous forefficiently utilizing the computer resources of a cluster. For example,the supervisor controller 808 can be configured to analyze the nodes ofa cluster in order to determine how to best create a virtual cluster.

In an embodiment, the supervisor controller 808 created the firstvirtual cluster 824 because the supervisor controller 808 determine thatat the time there were three nodes having excess CPU capacity of 50%.The supervisor controller 808 can be configured to determine that theclient-submitted job requires 150% of CPU capacity. Accordingly, thesupervisor controller 808 can be configured to create the first virtualcluster 824 in order to satisfy the job requirement of the client 802.However, at another point in time, the supervisor controller 808 can beconfigured to determine that two additional nodes became free such that75% of the CPU capacity on each of the nodes was available. In anembodiment, the supervisor controller 808 can be configured to determinethat it is more efficient for processing a particular job using twonodes as opposed to processing the job over three nodes. For example,the use of two nodes can be faster for processing jobs. The sharing ofdata over three nodes requires more time than the sharing of databetween two nodes. Accordingly, the supervisor controller 808 can beconfigured to dynamically create a new first virtual cluster 826comprising node 4, 816, and node 5, 818, wherein each node can allocate75% of the CPU capacity of each node to processing the job from theclient 802.

FIG. 8C is a block diagram depicting a high-level overview of anembodiment of virtual clusters. Similar to FIG. 8A and 8B, the client802 can be configured to submit jobs to the master node 804. In anembodiment, the supervisor controller 808 can be configured to createvirtual clusters wherein certain nodes are part of one or more clusters.For example, the supervisor controller 808 can be configured to createthe first virtual cluster 836 comprising node 1, 824, node 2, 826, andnode 3, 824. With respect to node 1 and 2, the supervisor controller 808can be configured to designate 100% of the CPU capacity for these nodesto be dedicated for the first virtual cluster 836.

With respect to node 3, the supervisor controller 808 can be configuredto designate only 50% of the CPU capacity of this node for the firstvirtual cluster 836. The supervisor controller 808 can be configured togenerate a second virtual cluster 838 comprising nodes 3, 828, node 4,830, node 5, 932, and node 6, 834. In an embodiment, the supervisorcontroller 808 can be configured to designate that only 50% of the CPUcapacity of node 3 should be dedicated to the second virtual cluster838. With respect to nodes 4, 5, and 6, the supervisor controller 808can be configured to designated 100% of the CPU capacity for these nodesto the second virtual cluster 838.

FIG. 8D is a block diagram depicting a high-level overview of anembodiment of virtual clusters. In an embodiment, the supervisorcontroller 808 can be configured to generate any number of virtualclusters based on the nodes of a single physical cluster. For example,the supervisor controller 808 can be configured to generate threevirtual clusters. The supervisor controller 808 can be configured togenerate a first virtual cluster comprising node 836, 838, and 840. Thesupervisor controller 808 can be configured to designate the node 836 todedicate 80% of the CPU capacity to the first virtual cluster whiledesignating only 10% of the CPU capacity of node 838 to the firstvirtual cluster and dedicating 50% of the CPU capacity of the node 840to the first virtual cluster.

The supervisor controller 808 can be configured to generate a secondvirtual cluster comprising nodes 836, 838, 840, 842, and 844. Thesupervisor controller 808 can be configured to designate only 20% of theCPU capacity of the node 836 to the second virtual cluster whilededicating 90% of the CPU capacity of the node 838 to the second virtualcluster and dedicating 50% of the CPU capacity of the node 840 to thesecond virtual cluster and dedicating 100% of the CPU capacities of thenodes 842 and 844 to the second virtual cluster. The supervisorcontroller 808 can be configured to generate a third virtual clustercomprising node 846. The supervisor controller 808 can be configured todesignate that 100% of the CPU capacity of the node 846 be dedicated tothe third virtual cluster.

FIG. 8E is a block diagram depicting a high-level overview of anembodiment of virtual clusters. Similar to FIG. 8A, the client 802 canbe configured to submit jobs for processing on a cluster to a masternode 804. In an embodiment, the supervisor controller 808 can not onlyallocate CPU capacity on particular nodes to specific virtual clusters,but also the supervisor controller 808 can be configured to dedicateother computing resources on the node to specific virtual clusters. Forexample, the supervisor controller 808 can be configured to dedicate 50%of RAM usage on node 848 to the first virtual cluster and to the secondvirtual cluster.

In addition to dedicating computer resources at particular nodes tospecific virtual clusters, the supervisor controller 808 can also beconfigured to dedicate computer resources of other devices in thecluster or connected to the cluster to specific virtual clusters. Forexample, the supervisor controller 808 can be configured to dedicate 30%of the switch utilization of a first switch 860 to the first virtualcluster. Similarly, the supervisor controller 808 can be configured toallocate 0% of a switch usage of a second switch 862.

FIG. 9 is a flow chart depicting an embodiment of a process forprocessing jobs using a virtual cluster. At block 902 the process canbegin with a job being received at block 904. The system can beconfigured to determine whether the submitted job is designated to beprocessed by a virtual cluster. If the determination is yes, then atblock 908, the job tracker or other management software, and/orsupervisor controller can be configured to divide the job into varioussub-jobs for assignment to nodes in the virtual cluster designated bythe system.

At block 910, the job tracker or other management software, and/orsupervisor controller can be configured to determine which node(s) inthe virtual cluster to assign the task or otherwise put the task in aqueue for later processing. At block 912, the supervisor and/or agentcontrollers can be configured to determine which task in the queueshould be assigned to nodes outside the virtual cluster. For example,the supervisor and/or the agent controllers can be configured todetermine that the job is a high priority job and therefore should beprocessed as soon as possible using other nodes outside the virtualcluster.

In another example, the supervisor and/or the agent controllers can beconfigured to determine that other nodes outside of the virtual clusterhave computer resources available for processing job(s). Accordingly,the supervisor and the agent controllers can be configured to assignsub-jobs in the queue to available nodes outside the virtual cluster atblock 914. If at block 912, the supervisor and/or the agent controllerdetermine that a sub-job in the queue should not be assigned to marriageoutside of the virtual cluster, the process can return to block 910where the job tracker or other management software, and the supervisorcontroller can be configured to determine which node in the virtualcluster to assign a sub-job.

If the determination at block 906 is no, then at block 914 the jobtracker or other management software, and supervisor controller can beconfigured to divide the job submission into sub-jobs for assignment tonodes outside the virtual cluster. At block 916, the job tracker and/orsupervisor controller can be configured to determine which availablenodes outside the virtual cluster to assign the sub-jobs, or otherwiseput the sub-job in a queue for later processing.

FIG. 10 is a flowchart depicting an embodiment of a process forprocessing jobs using a virtual cluster. The process can begin at 1002with receiving a job submission at block 1004. At block 1006, the systemcan be configured to determine whether a job submission is designated tobe processed by a virtual cluster. At block 1008, the system can beconfigured to determine the resources necessary to process the job basedon the client specified performance goals. At block 1010, the system canbe configured to generate and/or identify a virtual cluster based on therequired resource necessary for processing the job and/or based on theresources available in the nodes of the cluster and/or based on thespecified performance goals of the user/client.

At block 1012, the system can be configured to assign sub-jobs to thenodes in the created virtual cluster and can be configured to add theassigned sub-jobs to a queue for later processing. At block 1014, thesystem can be configured to optionally determine which sub-jobs in thequeue should be assigned to nodes outside the virtual cluster. At block1014, the system can be configured to optionally return to block 1010where a virtual cluster is identified for processing the jobs in thequeue.

FIG. 11 is a flowchart depicting an embodiment of a process forprocessing jobs using job groups. The process can begin at block 1102with receiving a job submission at block 1104. The system can beconfigured to determine at block 1106 a job group type based on the jobsubmission and/or the job submission requirements. At block 1108, thesystem can be configured to allocate based on the job groupidentification CPU capacity, RAM capacity, disk I/O capacity, and/ornetwork capacity. At block 1110, the job tracker and/or the supervisorcontroller can be configured to divide the job submission into sub-jobsfor assignment to designated nodes with designated resource allocations.At block 1112, the supervisor and/or the agent controller can beconfigured to monitor the nodes to determine if resource allocations aresufficient for the jobs to be processed based on the job groupdesignation. If the determination is yes, then system can be configuredto optionally return to block 1112 to continue monitoring theacceptability of the resource allocation. If the determination at block1112 is no, then the system can be configured to return to block 1108 inorder to allocate nodes with specific CPU capacities, RAM capacities,disk I/O capacity, and/or network capacities for processing the jobbased on the designated job group.

FIG. 12 is a flowchart depicting an embodiment of a process formonetizing resources on a computer cluster, for example, sellingcomputer resources on a cluster to customers. In an embodiment, theselling of computer resources is different from selling virtual machinesbecause the latter requires that whole virtual machines be sold tocustomers whereas the former requires only the computer resources to besold to the customer. The selling of computer resources can be moreefficient and/or more cost effective for the customer and/or theoperator of the cluster.

One of the ordinary skill in the art will appreciate that the monetizingor selling of computer resources need not require the actual sale ofcomputer resources for currency but rather can also be applied to thecontext where resources are accounted for through intra-companybudgeting. For example, the system can be configured to provide computerresources of the cluster to departments (for example, legal department,marketing department, human resources department, and the like) of acompany based on a service plan level assigned to the department. In anembodiment, the service plan level assigned to a company department canequate to a budgetary accounting to the department for the company'scosts in operating and maintaining the computer cluster.

The process can begin at block 1202 with the accessing of a jobsubmission at block 1204. The system can be configured to determine atblock 1206 a customer service plan level for the particular jobsubmission. The system can be configured to determine the customerservice plan level by accessing the customer database/service planlevels database 1220. Customers can select service level requirementsand/or plans at block 1218 where such data is stored in the customerdatabase/service plans levels database 1220.

At block 1208, the system can be configured to determine the resourcesnecessary to complete the job submission. At block 1212, the job trackeror are other management software and/or the supervisor controller can beconfigured to divide the job submission into some jobs for assignment todesignated nodes with designated resource allocations based on theservice plan level of the customer. At block 1212, the system can beconfigured to determine if resources are available to process thesub-job based on the service plan level of the customer. If thedetermination is yes, then at block 1214 the system can be configured toassign the sub-job to an available node based on the service plan levelof the customer. If the determination at block 1212 is no, then thesystem can be configured to add the sub-job to a queue for processingafter a node becomes available based on the service plan level of thecustomer. In an embodiment, the customer selection of service levelrequirements can be specified differently for a particular job from acustomer. For example, a customer may specify a higher service level foran urgent job than for that customer's usual jobs, and if meeting thathigher service level requires additional resources, the system can beconfigured to charge the customer more for running that job or sub-jobthan if the customer had received the usual service level.

FIG. 13 is a block diagram depicting a high level overview of anembodiment of a computer cluster comprising heterogeneous nodes. In anembodiment, client 1302 can submit jobs for processing on cluster 1301to master node 1304. In an embodiment, the master node 1304 can comprisea job tracker or other management software 1306 that can be configuredto receive job submissions from the client 1302. The job tracker orother management software 1306 can be configured to analyze the jobsubmission and/or be configured to divide the job into sub-jobs forprocessing by various nodes 1310, 1324 in the cluster 1301.

In an embodiment, the nodes 1310, 1324 can comprise a task tracker forother node manager 1312, 1326 and can comprise an agent controller 1314,1328. The task tracker or other node manager 1312, 1326 can beconfigured to receive and/or tract the sub-job from the job tracker orother management software 1306. In an embodiment, the agent controller1314, 1328 can be configured to also track and monitor the processing ofthe sub-job by the node. In an embodiment, the agent controller 1314,1328 can also be configured to determine the total available computerresources that are provided for by a particular node 1310, 1324. Forexample, the node 1310 can provide a total of 100 units of CPU capacity1316, 100 units of RAM capacity 1318, 100 units of network capacity1320, and 100 units of IO capacity 1322.

By comparison, the node 1324 can provide 200 units of CPU capacity 1330,300 units of RAM capacity 1332, 250 units of network capacity 1334, and40 units of IO capacity 1336. In determining the total availablecomputer resources provided for by a particular node, the agentcontroller 1314, 1328 can be configured to transmit such data to thesupervisor controller 1308 in order for the supervisor controller todetermine a global awareness of the total amount of computer resourcesavailable in the cluster.

In an embodiment, the agent controller 1314, 1328 can also be configuredto determine the amount of computer resources utilized by the jobs beingoperated on by a particular noted. For example, the agent controller1314, 1328 an be configured to determine that a particular job isutilizing -50 units of CPU capacity 1316 on node 1310. Further, theagent controller 1328 can be configured to determine that a second jobis utilizing 100 units of CPU capacity 1330 on node 1324. The agentcontroller 1314, 1328 can be configured to transmit the CPU usage datato the supervisor controller 1308.

In an embodiment the agent controller 1314, 1328 can be configured todetermine the amount of computer resources that are not being utilizedat a particular node. For example, the agent controller 1314 can beconfigured to determine that 50 units of CPU capacity 1316 are not beingutilized by the job being operated on by node 1310. Similarly, the agentcontroller 1328 can be configured to determine that 100 units of CPUcapacity 1330 are not being utilized by the second job that is beingoperated on by node 1324. The agent controller 1314, 1328 can beconfigured to transmit the available unused computer resource data tothe supervisor controller 1308. In an embodiment, the supervisorcontroller 1308 and/or the agent controller 1314, 1328 can be configuredto allocate additional resources to existing jobs being operated on bynodes in the cluster or can be configured to allocate additional jobs orsub-jobs to the nodes in order to fully utilize the available computerresources that are provided for by the nodes.

As illustrated in FIG. 13, node 1310 and node 1324 provide differingamounts of computer resources. Accordingly the node 1310 and the node1324 are not homogeneous but rather together make up a heterogeneouscluster because the cluster is said to have different kinds of computerservers that offer varying amounts of computer resources. By trackingthe amount of available computer resources not being utilized by currentjobs on the nodes, the agent controller 1314, 1328 can be configured toenable the efficient utilization of heterogeneous clusters.

In an embodiment, the agent controller 1314, 1328 in conjunction withthe supervisors controller 1308 can be configured to fully utilize theavailable computer resources being offered by the heterogeneous clusterby allocating as many jobs to each of the different nodes based on eachof the nodes available computer resources that can be utilized forprocessing additional jobs.

FIG. 14 is a flowchart depicting an embodiment of a process forprocessing jobs utilizing a heterogeneous computer clusters. The processcan begin at block 1402 with the accessing of a job submission at block1014. At block 1406, the job tracker or other management software,and/or supervisor controller can be configured to divide the jobsubmission into tasks or sub-jobs for assignment to a first node and asecond node. At block 1408, the agent controls operating on the firstnode and the second node can be configured to determine if additionalcomputer resources are available for processing additional jobs. If thedetermination at node 1 is that no computer resources are available atnode 1 for processing additional jobs, then at block 1410 the agentcontroller can be configured to loop back to block 1408 to continue tocheck whether the node 1 has additional resources available forprocessing other jobs because the utilization of computer resources onany particular node is continuously changing.

If the determination at node 2 is that there are additional resourcesavailable on node 2 for processing additional jobs, then the agentcontroller operating under node 2 can be configured to transmit theresource availability data of node 2 to the supervisor controller and/orjob tracker or other management software operating on the master node.At block 414, the job tracker or other management software, and/or thesupervisor controller can be configured to assign additional tasks forsub-jobs to the second node. At block 414 the agent controller can beconfigured to loop back to block 1408 to continuously check whetheradditional resources become available for processing other jobs. Thisprocess can enable the full utilization of heterogeneous clustersbecause the system continuously checks each node to determine whetheradditional computer resources are available for processing additionaljobs.

FIG. 15 is a schematic diagram illustrating an embodiment of utilizingjob histories for improving resource allocation of a computer cluster.The top half of FIG. 15 illustrates a standard allocation of sub-jobsand/or tasks. The bottom half of FIG. 15 illustrates a dynamicallocation of sub-jobs and/or tasks based on job history data. In anembodiment, a first job has a historical resource utilization chartillustrated in chart 1508. As can be seen, the first job has at first ahigh resource utilization at the beginning stages of processing the joband then has a period of low resource utilization in the middle of theperiod and towards the end of the period the first job has a highresource utilization.

A second job comprises a resource utilization illustrated in chart 1510.At the start, the second job has a low resource utilization and towardsthe middle period of the job, there is a high resource utilization andtowards the end of the job there is very low resource utilization. Atypical cluster system would assign job 1 to a first node and wouldassign job 2 to a second node. Chart 1502 illustrates the resourceutilization of job 1 versus the overall resources available forallocation at node 1. Chart 1504 illustrates the resource utilization ofjob 2 relative to the overall resources available for allocation at node2. As illustrated in charts 1502 and 1504, there are significant periodswhere the computer resources of node 1 and node 2 are underutilizedbecause of the low resource utilization periods of job 1 and job 2.Accordingly it would be advantageous to operate jobs 1 and 2 on a singlenode in order to have full utilization of a particular node.

In an embodiment, the resource monitoring and allocation systemsdisclosed herein can be configured to allow for more efficientutilization of nodes by analyzing the historical resource utilization ofjobs and predicting the utilization rates of particular jobs in order tocombine certain jobs with other jobs that would allow for more efficientutilization of the resources available for allocation at a particularnode. For example, as illustrated in chart 1512 and 1514, job 1comprises a low resource utilization during the middle of the period forcompleting the job while job 2 has a high resource utilization rateduring the middle period of completing the job. Accordingly by sendingboth job 1 and job 2 to a single node, there can be more efficientoverall use of the computer resources available for allocation at node 1as illustrated in chart 1506.

FIG. 16 is a flowchart depicting an embodiment of a process forgenerating reports relating to hardware modifications and/or additionsto a computer cluster. The process can begin at block 1602 by receivinga job submission at block 1604. At block 1606, the supervisor controllercan be configured to determine the resources necessary to process a jobbased on client specified performance goals. At block 1608, thesupervisor controller can be configured to determine resources and/ornodes available to process the job. At block 1610, the job tracker orother management software can be configured to assign the sub-jobs toavailable nodes. In an embodiment, the supervisor controller can beconfigured to designate the allocation of computer resources for eachsub-job at each node. At block 1612, the agent controller can beconfigured to determine periodically or continuously the status and/orresource utilization of each sub-job at each node.

At block 1614, the agent controller and/or the supervisor controller canbe configured to identify resource limitation bottlenecks in the clusterbased on the determining of the status and/or resource utilization atthe various nodes in the cluster. At block 1616, the supervisorcontroller can be configured to generate a report listing resourcelimitation bottlenecks and/or hardware modifications and/or additions tomitigate bottlenecks in the cluster. At block 1616, the system can beconfigured to optionally loop back to block 1612 in order for the agentcontroller to periodically or continuously determine the status and/orresource utilization of each sub-job at each node.

FIG. 17 is a flowchart depicting an embodiment of a process forgenerating reports relating to resource reallocation on a computercluster. The process can begin at block 1702 by receiving a jobsubmission and at least one of a user identifier, job group, department,user group, or the like at block 1704. At block 1706, the supervisorcontroller can be configured to determine the resources necessary toprocess the job based on the client specified performance goals. Atblock 1708, the supervisor controller can be configured to determine theavailable resources and/or available nodes for processing the job. Atblock 1710, the job tracker or other management software or thesupervisor controller can be configured to assign sub-jobs to theavailable nodes.

At block 1710, the supervisor controller can be configured to designatethe allocation of resources for each sub-job at each node. At block1712, the agent controller can be configured to determine periodicallyor continuously the status and/or resource utilization of each sub-jobat each node. At block 1714, the supervisor controller can be configuredto identify resource limitation bottlenecks in the cluster based on thedetermining of the status and/or resource utilization of the sub-jobs atthe various nodes in the cluster. At block 1716, the supervisorcontroller can be configured to generate a report listing the resourcelimitation bottlenecks and/or at least one of the user identifiers, jobgroups, departments, user groups, or the like that is causing thebottlenecks. At block 1716, the system can be configured to loop back toblock 1712 in order for the agent controller to determine periodicallyor continuously the status and/or resource utilization of each sub-jobat each node.

FIG. 17A is a flowchart depicting an embodiment of a process fordetermining resource reallocation levels for application to jobs orsub-jobs. In an embodiment, the system can be configured to select asubset of tasks and/or sub-jobs of a particular job and tweak theresource allocation settings or configurations for the selected subsetof tasks and/or sub-jobs in order to discover how the tasks and/orsub-jobs react to the resource allocation settings. The system can beconfigured to apply different resource allocation settings orconfigurations to different subsets in order to determine the bestresource allocation settings for applying to particular sub-jobs. Forexample, with respect to the java virtual machine heap setting, thesystem can be configured to set the java virtual machine heap setting toaggressively return unused memory. The system can be configured tomonitor the performance characteristics of the sub-jobs based theforegoing setting. The system can be configured to use the resultinginformation to apply better control for future tasks or sub-jobs of thecurrent job or the future instances of the job.

Similarly, the system can be configured to determine the actual currentcapacity of a resource, such as disk I/O capacity or network capacity bydynamically adjusting threshold levels for access to these resources byvarious tasks or sub-jobs. For example, the system can be configured toincrease the network bandwidth requested by all tasks or sub-jobs (addedtogether) over the course of several time intervals until the networkstops providing the extra requested bandwidth, then assuming thatobserved maximum bandwidth provided is the currently availablebandwidth. The system can be configured to repeat this processcontinuously so that each node maintains an estimate of the availablemaximum capacity for each resource.

With reference to FIG. 17A, in an embodiment, the process can begin atblock 1718 with the system receiving a job submission at block 1720. Atblock 1722, the system can be configured to divide the job into aplurality of sub-jobs. At block 1724, the system can be configured toselect one or more subsets of the sub-jobs for applying experiments ofresource allocations to determine which resource allocation levels yieldthe best performance characteristics for the particular type of sub-jobsat issue. At block 1726, the system can be configured to apply variousresource allocation levels to different subsets of sub-jobs. At block1728, the system can be configured to monitor performancecharacteristics of sub-jobs in the various subsets based on the appliedresource allocation levels. At block 1730, the system can be configuredto determine which resource allocation levels yield the best performancecharacteristics for the sub-job type. At block 1732, the system can beconfigured to store the resource allocation level that yield the bestperformance characteristics for future application to similar sub-jobtypes or other sub-jobs that are part of the overall job.

FIG. 18 is a block diagram depicting a high level overview of anembodiment of a computer cluster comprising a dynamic monitoring and/orresource allocation system. In an embodiment, the client 1802 can submitjobs to the master node 1804 in order to have the job processed by thecluster 1801. The master node 1804 can comprise a management software1806 and can comprise a supervisor controller 1808. In an embodiment,the management software 1806 can be configured to analyze the jobreceived from the client 1802 and divide the job into various sub-jobsfor processing by the various nodes in the cluster. The cluster 1801 cancomprise a plurality of nodes 1822, 1842.

In an embodiment, the management software 1806 can be configured to sendsub-jobs 1820, 1818, 1838, 1840 to the various nodes 1822, 1842 forprocessing. In an embodiment, the other tracking software 1824, 1844 canbe configured to receive the sub-job from the master node in order forthe sub-jobs to be processed on the nodes. In an embodiment, the agentcontroller 1826, 1846 can be configured to track the progress of thesub-jobs that are being processed by the nodes and can be configured todetermine the resource allocation usage of each of the jobs running oneach of the nodes.

In an embodiment the agent controller 1826, 1846 can be configured totransmit the resource utilization data to the supervisor controller 1808that operates in the master node 1804. The supervisor controller 1808and/or the agent controller 1826, 1846 can be configured to determinewhether the resource allocation of a particular job on a particular nodeshould be reduced or increased or remain the same. In an embodiment, theagent controller 1826, 1846 can be configured to generate instructionsfor processing at the kernel, the process, or other module 1828, 1848 inorder to reduce, increase, or keep the resource allocation for theparticular sub-job at a particular node.

In an embodiment, the nodes 1822, 1842 can be configured to run othersoftware applications including but not limited to web server 1830,1850, database 1832, virtual machine 1852, impala query engine 1834,database query manager 1854, and other software applications 1836, 1856.In an embodiment, the agent controller 1826, 1846 can be configured todetermine the resource utilization of each of the software applicationrunning on the various nodes. In an embodiment, the agent controller1826, 1846 can be configured to transmit the resource utilization of thesoftware applications operating on each of the nodes to the supervisorcontroller 1808. The agent controller 1826, 1846 and/or the supervisorcontroller 1808 can be configured to determine that the resources beingutilized by a particular software application on a particular nodeshould be reduced, increased, or remain the same.

In an embodiment, the cluster 1801 can comprise a network controller1812. The network controller 1812 can comprise a network router, anetwork switch, or the like. In an embodiment, the network controller1812 can comprise a agent controller 1810. The agent controller 1810 canbe configured to determine the resource utilization of the networkcontroller 1810 by certain nodes, jobs, sub-jobs, or applications. In anembodiment, the agent controller 1810 can be configured to transmit theresource utilization data to the supervisor controller 1808. Thesupervisor controller 1808 and/or the agent controller 1810 can beconfigured to reallocate the use of resources provided for by thenetwork controller 1810 for certain nodes, jobs, sub-jobs, and/orapplications.

In an embodiment, the cluster 1801 can be coupled or connected to anexternal resource 1816. The external resource 1816 can include but isnot limited to external databases, data extraction/transformation tools,web services, and the like. In an embodiment, the external resource 1816can comprise an agent controller. The agent controller can be configuredto determine the usage of resources of the external resource 1816 bynodes, jobs, sub-jobs, and/or applications. In an embodiment, the agentcontroller can be configured to transmit the resource utilization datato the supervisor controller 1808. In an embodiment, the supervisorcontroller 1808 and/or the agent controller 1814 can be configured todetermine whether the resource utilization of the external resource 1816by particular nodes, jobs, sub-jobs, and/or applications on a particularnode should be reduced, increased, or remained the same.

Computer System

In some embodiments, the systems, processes, and methods described aboveare implemented using a computing system, such as the one illustrated inFIG. 19. The example computer system 1902 is in communication with oneor more computing systems 1920 and/or one or more data sources 1922 viaone or more networks 1918. While FIG. 19 illustrates an embodiment of acomputing system 1902, it is recognized that the functionality providedfor in the components and modules of computer system 1902 may becombined into fewer components and modules, or further separated intoadditional components and modules.

Dynamic Resource Monitoring/Allocation Module

The computer system 1902 includes a dynamic resourcemonitoring/allocation module 1914 that carries out the functions,methods, acts, and/or processes described herein. The dynamic resourcemonitoring/allocation module 1914 is executed on the computer system1902 by a central processing unit 1910 discussed further below.

In general the word “module,” as used herein, refers to logic embodiedin hardware or firmware or to a collection of software instructions,having entry and exit points. Modules are written in a program language,such as JAVA, C or C++, or the like. Software modules may be compiled orlinked into an executable program, installed in a dynamic link library,or may be written in an interpreted language such as BASIC letters,PERL, LUA, or Python. Software modules may be called from other modulesor from themselves, and/or may be invoked in response to detected eventsor interruptions. Modules implemented in hardware include connectedlogic units such as gates and flip-flops, and/or may includeprogrammable units, such as programmable gate arrays or processors.

Generally, the modules described herein refer to logical modules thatmay be combined with other modules or divided into sub-modules despitetheir physical organization or storage. The modules are executed by oneor more computing systems, and may be stored on or within any suitablecomputer readable medium, or implemented in-whole or in-part withinspecial designed hardware or firmware. Not all calculations, analysis,and/or optimization require the use of computer systems, though any ofthe above-described methods, calculations, processes, or analyses may befacilitated through the use of computers. Further, in some embodiments,process blocks described herein may be altered, rearranged, combined,and/or omitted.

Computing System Components

The computer system 1902 includes one or more processing units (CPU)1910, which may include a microprocessor. The computer system 1902further includes a memory 1912, such as random access memory (RAM) fortemporary storage of information, a read only memory (ROM) for permanentstorage of information, and a mass storage device 1904, such as a harddrive, diskette, or optical media storage device. Alternatively, themass storage device may be implemented in an array of servers.Typically, the components of the computer system 1902 are connected tothe computer using a standards based bus system. The bus system can beimplemented using various protocols, such as Peripheral ComponentInterconnect (PCI), Micro Channel, SCSI, Industrial StandardArchitecture (ISA) and Extended ISA (EISA) architectures.

The computer system 1902 includes one or more input/output (I/O) devicesand interfaces 1908, such as a keyboard, mouse, touch pad, and printer.The I/O devices and interfaces 1908 can include one or more displaydevices, such as a monitor, that allows the visual presentation of datato a user. More particularly, a display device provides for thepresentation of GUIs as application software data, and multi-mediapresentations, for example. The I/O devices and interfaces 1908 can alsoprovide a communications interface to various external devices. Thecomputer system 1902 may include one or more multi-media devices 1906,such as speakers, video cards, graphics accelerators, and microphones,for example.

Computing System Device/Operating System

The computer system 1902 may run on a variety of computing devices, suchas a server, a Windows server, and Structure Query Language server, aUnix Server, a personal computer, a laptop computer, and so forth. Inother embodiments, the computer system 1902 may run on a clustercomputer system, a mainframe computer system and/or other computingsystem suitable for controlling and/or communicating with largedatabases, performing high volume transaction processing, and generatingreports from large databases. The computing system 1902 is generallycontrolled and coordinated by an operating system software, such asz/OS, Windows 95, Windows 98, Windows NT, Windows 2000, Windows XP,Windows Vista, Windows 7, Linux, UNIX, BSD, SunOS, Solaris, or othercompatible operating systems, including proprietary operating systems.Operating systems control and schedule computer processes for execution,perform memory management, provide file system, networking, and I/Oservices, and provide a user interface, such as a graphical userinterface (GUI), among other things.

Network

The computer system 1902 illustrated in FIG. 19 is coupled to a network1918, such as a LAN, WAN, or the Internet via a communication link 1916(wired, wireless, or a combination thereof). Network 1918 communicateswith various computing devices and/or other electronic devices. Network1918 is communicating with one or more computing systems 1920 and one ormore data sources 1922. The dynamic resource monitoring/allocationmodule 1914 may access or may be accessed by computing systems 1920and/or data sources 1922 through a web-enabled user access point.Connections may be a direct physical connection, a virtual connection,and other connection type. The web-enabled user access point may includea browser module that uses text, graphics, audio, video, and other mediato present data and to allow interaction with data via the network 1918.

The output module may be implemented as a combination of an all-pointsaddressable display such as a cathode ray tube (CRT), a liquid crystaldisplay (LCD), a plasma display, or other types and/or combinations ofdisplays. The output module may be implemented to communicate with inputdevices 1908 and they also include software with the appropriateinterfaces which allow a user to access data through the use of stylizedscreen elements, such as menus, windows, dialogue boxes, tool bars, andcontrols (e.g., radio buttons, check boxes, sliding scales, and soforth). Furthermore, the output module may communicate with a set ofinput and output devices to receive signals from the user.

Other Systems

The computing system 1902 may include one or more internal and/orexternal data sources (e.g., data sources 1922). In some embodiments,one or more of the data repositories and the data sources describedabove may be implemented using a relational database, such as DB2,Sybase, Oracle, CodeBase, and Microsoft® SQL Server as well as othertypes of databases such as a flat-file database, an entity relationshipdatabase, and object-oriented database, and/or a record-based database.

The computer system 1902 also accesses one or more databases 1922. Thedatabases 1922 may be stored in a database or data repository. Thecomputer system 1902 may access the one or more databases 1922 through anetwork 1918 or may directly access the database or data repositorythrough I/O devices and interfaces 1908. The data repository storing theone or more databases 1922 may reside within the computer system 1902.

Conditional language, such as, among others, “can,” “could,” “might,” or“may,” unless specifically stated otherwise, or otherwise understoodwithin the context as used, is generally intended to convey that certainembodiments include, while other embodiments do not include, certainfeatures, elements and/or steps. Thus, such conditional language is notgenerally intended to imply that features, elements and/or steps are inany way required for one or more embodiments or that one or moreembodiments necessarily include logic for deciding, with or without userinput or prompting, whether these features, elements and/or steps areincluded or are to be performed in any particular embodiment. Theheadings used herein are for the convenience of the reader only and arenot meant to limit the scope of the inventions or claims.

Additional Embodiments

Although this invention has been disclosed in the context of certainpreferred embodiments and examples, it will be understood by thoseskilled in the art that the present invention extends beyond thespecifically disclosed embodiments to other alternative embodimentsand/or uses of the invention and obvious modifications and equivalentsthereof. Additionally, the skilled artisan will recognize that any ofthe above-described methods may be carried out using any appropriateapparatus. Further, the disclosure herein of any particular feature,aspect, method, property, characteristic, quality, attribute, element,or the like in connection with an embodiment may be used in all otherembodiments set forth herein. Thus, it is intended that the scope of thepresent invention herein disclosed should not be limited by theparticular disclosed embodiments described above.

1. (canceled)
 2. (canceled)
 3. (canceled)
 4. (canceled)
 5. (canceled) 6.(canceled)
 7. (canceled)
 8. (canceled)
 9. (canceled)
 10. (canceled) 11.(canceled)
 12. (canceled)
 13. (canceled)
 14. (canceled)
 15. (canceled)16. (canceled)
 17. (canceled)
 18. (canceled)
 19. (canceled) 20.(canceled)
 21. A non-virtual computer cluster system, the systemcomprising: a management computing device comprising one or morecomputer readable storage devices configured to store a plurality ofcomputer executable instructions and one or more hardware computerprocessors in communication with the one or more computer readablestorage devices and configured to execute the plurality of computerexecutable instructions in order to cause the management computingdevice to coordinate processing of a plurality of sub-jobs for anoverall job; and a plurality of computer system nodes configured toperform processing of the plurality of sub-jobs for the overall job,each of the plurality of computer system nodes comprising one or morecomputer readable storage devices configured to store a plurality ofcomputer executable instructions and one or more hardware computerprocessors in communication with the one or more computer readablestorage devices and configured to execute the plurality of computerexecutable instructions in order to cause each of the plurality ofcomputer system nodes to: monitor utilization of system resources of thecomputer system node for processing one or more of the plurality ofsub-jobs, the system resources comprising at least one of CPU,electronic storage input/output, network, and memory of the computersystem node; and periodically report the monitored system resourceutilization to the management computing device, wherein the managementcomputing device is a separate computing device from each of theplurality of computer system nodes, and wherein the management computingdevice is configured to coordinate processing of the plurality ofsub-jobs by the plurality of computer system nodes based at least inpart on the monitored system resource utilization of the plurality ofcomputer system nodes.
 22. The non-virtual computer cluster system ofclaim 21, wherein the management computing device is further configuredto determine progress for completing the overall job by tracking theplurality of sub-jobs being processed by the plurality of computersystem nodes.
 23. The non-virtual computer cluster system of claim 21,wherein the management computing device is further configured todynamically increase processing of higher priority sub-jobs.
 24. Thenon-virtual computer cluster system of claim 21, wherein the managementcomputing device operates a master node of the computer cluster.
 25. Thenon-virtual computer cluster system of claim 21, wherein the managementcomputing device is further configured to dynamically adjust systemresource allocations for each sub-job on the plurality of computersystem nodes.
 26. The non-virtual computer cluster system of claim 25,wherein the dynamic adjustment of system resource allocations is basedat least in part on the periodic reporting from the plurality ofcomputer system nodes.
 27. The non-virtual computer cluster system ofclaim 26, wherein the dynamic adjustment of system resource allocationscomprises allocating additional central processing unit resources to asub-job operating on a computer system node based at least in part onthe management computing device determining that a total amount ofunused central processing unit resources available at the computersystem node exceeds a threshold level.
 28. The non-virtual computercluster system of claim 27, wherein the dynamic adjustment of systemresource allocations comprises allocating additional networkingresources to a sub-job operating on a computer system node based atleast in part on the management computing device determining that atotal amount of unused networking resources available at the computersystem node exceeds a threshold level.
 29. The non-virtual computercluster system of claim 27, wherein the dynamic adjustment of systemresource allocations comprises allocating additional random-accessmemory resources to a sub-job operating on a computer system node basedat least in part on the management computing device determining that atotal amount of unused random-access memory resources available at thecomputer system node exceeds a threshold level.
 30. The non-virtualcomputer cluster system of claim 27, wherein the dynamic adjustment ofsystem resource allocations comprises allocating additional electronicstorage input-output resources to a sub-job operating on a computersystem node based at least in part on the management computing devicedetermining that a total amount of unused electronic storageinput-output resources available at the computer system node exceeds athreshold level.
 31. The non-virtual computer cluster system of claim21, wherein the management computing device is further configured toreceive goals specified by an operator of the computer cluster system,wherein the goals comprise a priority level for one or more of theplurality of sub-jobs.
 32. The non-virtual computer cluster system ofclaim 31, wherein the management computing device is configured toassign a first level of system resource allocations to a first sub-joband a second level of system resource allocations to a second sub-job,wherein the first level is greater than the second level, wherein thefirst sub-job comprises a first priority level and the second sub-jobcomprises a second priority level, wherein the first priority level isgreater than the second priority level.
 33. The non-virtual computercluster system of claim 31, wherein the goals specified by the operatorof the computer cluster comprises a user group preference for one ormore of the plurality of sub-jobs.
 34. The non-virtual computer clustersystem of claim 33, wherein the management computing device isconfigured to assign a first level of system resource allocations to afirst sub-job and a second level of system resource allocations to asecond sub-job, wherein the first level is greater than the secondlevel, wherein the first sub-job comprises a first user group preferenceand the second sub-job comprises a second user group preference, whereinsub-jobs associated with the first user group preference are processedwith greater priority than sub-jobs associated with the second usergroup preference.
 35. A computer-implemented method for coordinatingprocessing of a plurality of sub-jobs for an overall job by a pluralityof computer system nodes in a non-virtual computer cluster system, themethod comprising: monitoring, by each of the plurality of computersystem nodes of the computer cluster system, utilization of systemresources of each of the plurality of computer system nodes forprocessing one or more of the plurality of sub-jobs for the overall job,the system resources comprising at least one of CPU, electronic storageinput/output, network, and memory of the particular computer systemnode; periodically reporting, by each of the plurality of computersystem nodes of the computer cluster system, the monitored systemresource utilization to the management computing device of the computercluster system; receiving, by the management computing device, themonitored system resource utilization of each of the plurality ofcomputer system nodes of the computer cluster system; and coordinating,by the management computing device, processing of the plurality ofsub-jobs by the plurality of computer system nodes based at least inpart on the monitored system resource utilization of the plurality ofcomputer system nodes, wherein each of the plurality of computer systemnodes and the management computing device comprises a computer processorand an electronic storage medium.
 36. The computer-implemented method ofclaim 35 further comprising determining, by the management computingdevice, progress for completing the overall job by tracking theplurality of sub-jobs being processed by the plurality of computersystem nodes.
 37. The computer-implemented method of claim 35 furthercomprising receiving, by the management computing device, a prioritylevel for one or more of the plurality of sub-jobs from an operator ofthe computer cluster system.
 38. The computer-implemented method ofclaim 37 further comprising assigning, by the management computingdevice, a first level of system resource allocations to a first sub-joband a second level of system resource allocations to a second sub-job,wherein the first level is greater than the second level, wherein thefirst sub-job comprises a first priority level and the second sub-jobcomprises a second priority level, wherein the first priority level isgreater than the second priority level.
 39. The computer-implementedmethod of claim 35, further comprising dynamically adjusting, by themanagement computing device, system resource allocations for one or moreof the plurality of sub-jobs being processed by the plurality ofcomputer system nodes.
 40. The computer-implemented method of claim 39,wherein the dynamic adjustment of system resource allocations is basedat least in part on the periodic reporting from the plurality ofcomputer system nodes.